Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
Browse files

Merge branch 'services-r10' into master

commit 8723f18
Author: Andrew Liu <aliu@vmware.com>
Date:   Fri Apr 13 14:19:24 2012 +0800

    revendor base gem to include r9 commits

    Change-Id: Icdcabd36ceaad9fe17cc005d627dd3522e02164b

commit 7c893a9
Merge: 554be90 8d21b04
Author: Andrew Liu <aliu@vmware.com>
Date:   Fri Apr 13 14:15:07 2012 +0800

    Merge remote branch 'origin/services-r9' into r10

    Conflicts:
    	atmos/vendor/cache/vcap_services_base-0.1.8.gem
    	filesystem/vendor/cache/vcap_services_base-0.1.8.gem
    	mongodb/vendor/cache/vcap_services_base-0.1.8.gem
    	mysql/vendor/cache/vcap_services_base-0.1.8.gem
    	neo4j/vendor/cache/vcap_services_base-0.1.8.gem
    	postgresql/vendor/cache/vcap_services_base-0.1.8.gem
    	rabbit/vendor/cache/vcap_services_base-0.1.8.gem
    	redis/vendor/cache/vcap_services_base-0.1.8.gem
    	service_broker/vendor/cache/vcap_services_base-0.1.8.gem
    	vblob/vendor/cache/vcap_services_base-0.1.8.gem

    Change-Id: Iaac55526897acf42ed75fe3c08a96c1ca8b0687c

commit 554be90
Author: Andrew Liu <aliu@vmware.com>
Date:   Tue Apr 10 22:33:05 2012 +0800

    create new redis client in lock refresh thread

    Change-Id: I73f4efff1a7fda8bdc05b29eb216031137622fa8

commit 1d5dafe
Author: felixhoo <hhu@vmware.com>
Date:   Sat Apr 7 11:08:11 2012 -0600

    Invoke each run task in a fiber context

    Change-Id: Ibb7a167e6b5b4ea775d25bbddd7cbd702360e79e

commit 9468503
Author: Tang Rui <rtang@vmware.com>
Date:   Sun Apr 1 14:45:52 2012 -0700

    Fix capacity calculation for service migration

    Change-Id: I2befe4eed16a190e2128a8744a44a9f72bff57dc

commit 11c38be
Merge: bb3c024 f202332
Author: Frank Lu <flu@vmware.com>
Date:   Thu Apr 5 05:50:19 2012 +0000

    Merge "Parent role should not be blocked in disable_instance" into services-r10

commit bb3c024
Author: felixhoo <hhu@vmware.com>
Date:   Wed Apr 4 11:31:56 2012 -0600

    Add base gem to backupmanager

    Change-Id: I1c5bc77fb8f7196a7c29ed4f242753807e89a390

commit f202332
Author: Frank Lu <flu@vmware.com>
Date:   Sun Apr 1 09:14:39 2012 -0600

    Parent role should not be blocked in disable_instance

      1. Default user (Parent role) should not be blocked for it will be used in dump_instance
      2. Unit test update: kill long transaction should only work for normal user rather than default user and super user

    Change-Id: I032ac9060eef24f476d9d947c550ee952b0ca1a8

commit c5e63e1
Merge: b1ae681 c9d79a9
Author: Andrew Liu <aliu@vmware.com>
Date:   Tue Apr 3 05:08:59 2012 +0000

    Merge "add snapshot quota and delete snapshot job" into services-r10

commit c9d79a9
Author: Andrew Liu <aliu@vmware.com>
Date:   Wed Mar 14 22:18:27 2012 -0700

    add snapshot quota and delete snapshot job

    also update async job framwork:
    1. Unify job related namespace to VCAP::Services::Base::AsyncJob
    2. Update resque and resque-status gem
    3. Move common job logic into base.

    Change-Id: I2dac7d2d2b5ed20003926a607f4fdaf96096e33e

commit b1ae681
Author: Yang Yang <yangy@vmware.com>
Date:   Sun Apr 1 19:46:12 2012 -0700

    fixed a bug in base unit test:

    1. re-edited the MockErrorNode malformed message so that it can be encoded in json

    Change-Id: If337faccb5c4c2daf906039cbc1e18af99c4a6c0

commit 37dc696
Author: Chunjie <czhu@vmware.com>
Date:   Sat Mar 31 06:30:12 2012 -0700

    Add over provisioning support within varz.

    Change-Id: I276e5b99a593ca3cdb652d5c28ba0278ef622143

commit e637210
Merge: 38f9ce0 41132b6
Author: Chunjie <czhu@vmware.com>
Date:   Sat Mar 31 13:12:21 2012 +0000

    Merge "Add varz support for serialization data server." into services-r10

commit 38f9ce0
Merge: c9fcb39 9e5f01e
Author: Nicholas Kushmerick <nicholask@vmware.com>
Date:   Sat Mar 31 13:06:36 2012 +0000

    Merge "clean up unit test directory under /tmp" into services-r10

commit c9fcb39
Merge: 7acca15 308934c
Author: Nicholas Kushmerick <nicholask@vmware.com>
Date:   Sat Mar 31 13:05:27 2012 +0000

    Merge "neo4j unit test clean up" into services-r10

commit 308934c
Author: Cliff Wang <cliffwang@vmware.com>
Date:   Sat Mar 31 03:18:25 2012 -0600

    neo4j unit test clean up

    Change-Id: I97d22c3c7334d15b1ab71cd755f8840792e07912

commit 9e5f01e
Author: felixhoo <hhu@vmware.com>
Date:   Sat Mar 31 03:34:11 2012 -0600

    clean up unit test directory under /tmp

    Change-Id: Id1a7789dc74b3e9d156e8824f0a2292c9957de73

commit 7acca15
Author: figo <figof@mozy.com>
Date:   Sat Mar 31 00:16:00 2012 -0700

    add additional option for db_size_overhead to start pg nodewq

    Change-Id: I65b5c31c72e6c35832c105cc42376de980d604f5

commit c14b4d0
Author: Frank Lu <flu@vmware.com>
Date:   Tue Mar 13 02:51:25 2012 -0600

    backup/dump/import/restore/snapshot/serialize postgresql databases using parent role rather than superuser

         1. Use the default user (parent role) rather than the super user to backup/dump/import/restore/snapshot/serialize the provisioned database
         2. Add model.rb: list all DataMapper relative classes
         3. Prevent the long-time queries/transctions submmitted by parent roles and super users from being killed
         4. Filter the long-time queries/transactions by postgresql query rather than ruby statements
         5. Support wqsnapshot/serialization jobs
         6. Fix the issue that unit test will generate residual databases in postgresql

    Change-Id: Iefed934f5627f0874d4e789cac5a1d6329609f03

commit 50ac823
Merge: bb55965 402b92c
Author: Nicholas Kushmerick <nicholask@vmware.com>
Date:   Thu Mar 29 22:37:51 2012 +0000

    Merge "add quota and obj_limit configuration in vblob" into services-r10

commit bb55965
Author: figo <figof@mozy.com>
Date:   Thu Mar 29 06:00:24 2012 -0700

    psql quota should be max_db_size + db_size_overhead

    Change-Id: I544ad93e2a061a3637747e18c647636bdd677bb8

commit 41132b6
Author: Chunjie <czhu@vmware.com>
Date:   Thu Mar 29 00:25:12 2012 -0700

    Add varz support for serialization data server.

    The collector handler for serialization data server will be present soon.

    Change-Id: I0d5da6dbffb3cb655efb1d49d1199a0c551fd97f

commit 099b168
Merge: a2a39b2 49ecfe7
Author: figo <figof@mozy.com>
Date:   Thu Mar 29 06:46:58 2012 +0000

    Merge "make sure to close redis connection for varz" into services-r10

commit 49ecfe7
Author: figo <figof@mozy.com>
Date:   Wed Mar 28 01:57:20 2012 -0700

    make sure to close redis connection for varz

    Change-Id: I3e23a37b7fc1a034461398b46015b94523259af8

commit a2a39b2
Author: Andrew Liu <aliu@vmware.com>
Date:   Wed Mar 28 13:54:41 2012 +0800

    remove logging gem from services repository

    Change-Id: Ic9059f57964cb67d59414ceadf960c28f9a018f2

commit 402b92c
Author: SonicWang <wangs@vmware.com>
Date:   Tue Mar 27 10:44:32 2012 -0700

    add quota and obj_limit configuration in vblob

    Change-Id: Icd0d28007637aeb805c4c3614fd476ae1b03a89e

commit c5d3a2d
Merge: d2af5cf d3fb525
Author: SonicWang <wangs@vmware.com>
Date:   Tue Mar 27 17:16:41 2012 +0000

    Merge "remove max_memory parameter in vblob" into services-r10

commit d2af5cf
Author: Yang Yang <yangy@vmware.com>
Date:   Mon Mar 26 23:16:57 2012 -0700

    added a fake unit test for serialization data server for ci

    Change-Id: I7b62c4643dbdceb9c7d062b1cd593d74573468d8

commit 38c990e
Merge: 0b38eb1 9abcf88
Author: felixhoo <hhu@vmware.com>
Date:   Tue Mar 27 05:58:55 2012 +0000

    Merge "Add varz into backup manager" into services-r10

commit 0b38eb1
Merge: a25bf59 e0d4bd5
Author: Tang Rui <rtang@vmware.com>
Date:   Tue Mar 27 05:58:10 2012 +0000

    Merge "Kill epmd process used by rabbitmq server when finish unit test" into services-r10

commit a25bf59
Merge: 74288dd 89215eb
Author: Tang Rui <rtang@vmware.com>
Date:   Tue Mar 27 05:57:55 2012 +0000

    Merge "In redis provision, the node need wait for the start of the redis-server process" into services-r10

commit e0d4bd5
Author: Tang Rui <rtang@vmware.com>
Date:   Mon Mar 26 20:17:21 2012 -0700

    Kill epmd process used by rabbitmq server when finish unit test

    Change-Id: If76fc071f32cc9ffc429d9f8fb678e217cffd6dd

commit 89215eb
Author: Tang Rui <rtang@vmware.com>
Date:   Mon Mar 26 00:24:31 2012 -0700

    In redis provision, the node need wait for the start of the redis-server process

    Change-Id: I3722efc26846bf180ce6ead3c786bd9a03b2d979

commit 74288dd
Merge: ae48e35 bdcdffb
Author: felixhoo <hhu@vmware.com>
Date:   Mon Mar 26 20:09:48 2012 +0000

    Merge "Remove pid and add quota in worker config" into services-r10

commit d3fb525
Author: SonicWang <wangs@vmware.com>
Date:   Mon Mar 26 10:37:51 2012 -0700

    remove max_memory parameter in vblob

    Change-Id: I8963a7d88df83a96e1c9beef0360e0e537419405

commit 9abcf88
Author: felixhoo <hhu@vmware.com>
Date:   Mon Mar 26 01:20:31 2012 -0600

    Add varz into backup manager

    Change-Id: I777e5e141048202de5638a4930f33cbbe120cac7

commit ae48e35
Author: felixhoo <hhu@vmware.com>
Date:   Sun Mar 25 13:48:43 2012 -0600

    Fix time deviation of mysql backup

    Change-Id: I44b35a35a1adb1c7e0778766894046f3957d206b

commit bdcdffb
Author: felixhoo <hhu@vmware.com>
Date:   Sat Mar 24 12:58:14 2012 -0600

    Remove pid and add quota in worker config

    Change-Id: Ifdcd6e6e9fcc8df8f6db78775fa024990a757604

commit a3de99a
Merge: 493fc4c d45578d
Author: Tang Rui <rtang@vmware.com>
Date:   Fri Mar 23 03:53:27 2012 +0000

    Merge "Fix the redis warning: Could not cleanup instance, the reasons: ["ERR max number of clients reached"]" into services-r10

commit 493fc4c
Merge: 0e3e341 510817c
Author: Yang Yang <yangy@vmware.com>
Date:   Fri Mar 23 01:55:21 2012 +0000

    Merge "removed unused clover generator for coverage report for ci." into services-r10

commit 0e3e341
Merge: 0055e0e 2c5003c
Author: Andrew Liu <aliu@vmware.com>
Date:   Thu Mar 22 10:42:50 2012 +0000

    Merge "correct column name in user query" into services-r10

commit 510817c
Author: Yang Yang <yangy@vmware.com>
Date:   Thu Mar 22 13:54:47 2012 +0800

    removed unused clover generator for coverage report for ci.

    Change-Id: Ic1cc87da39406491ca6270533e8910a19970b6f1

commit d45578d
Author: Tang Rui <rtang@vmware.com>
Date:   Wed Mar 21 21:36:13 2012 -0700

    Fix the redis warning: Could not cleanup instance, the reasons: ["ERR max number of clients reached"]

    Change-Id: I2c9988c4337a42ca98409ca6a053985724cf6c5b

commit 0055e0e
Author: Tang Rui <rtang@vmware.com>
Date:   Wed Mar 21 20:35:52 2012 -0700

    Fix the bug that rabbit doesn't delete instance's logs when unprovision

    Change-Id: I13d84574c712b84080a881305b00c65ba9175eea

commit 2c5003c
Author: Andrew Liu <aliu@vmware.com>
Date:   Wed Mar 21 20:59:43 2012 +0800

    correct column name in user query

    Change-Id: Ib8ecd3f1ed11c5335d3f329b3a0c32cbf92dd019

commit 1d817df
Merge: 270634d dd67734
Author: Yang Yang <yangy@vmware.com>
Date:   Wed Mar 21 08:43:48 2012 +0000

    Merge "added coverage report generation feature for services" into services-r10

commit 270634d
Merge: f69403b ae50dc5
Author: Tang Rui <rtang@vmware.com>
Date:   Wed Mar 21 03:00:12 2012 +0000

    Merge "Each service node code need check datamapper object before destroy" into services-r10

commit dd67734
Author: Yang Yang <yangy@vmware.com>
Date:   Tue Mar 20 17:46:25 2012 +0800

    added coverage report generation feature for services

    1. reconstruct the feature to create coverage report generation using
      SimpleCov for the following services:
      base, atmos, filesystem, mongodb, mysql, neo4j, postgresql, rabbit,
      redis, serialization_data_server, service_broker, vblob.

    2. finished the unit test for using with the following rake command:
      bundle exec rake spec          # unit test without any report
      bundle exec rake spec:ci       # unit test with report for rcov and ci
      bundle exec rake spec:rcov     # unit test with report for rcov only

    3. added corresponding gem to the vendor/cache/ mainly for simplecov.

    Change-Id: I6a1cf069b38da65b5f5d4f3fc7ee6580ce7b587c

commit ae50dc5
Author: Tang Rui <rtang@vmware.com>
Date:   Sun Mar 18 22:35:55 2012 -0700

    Each service node code need check datamapper object before destroy

    Change-Id: Id5100305368b8d9525bc54914ebeab150c67dbfe

commit ff2a632
Merge: 68b0d4f ec3334f
Author: Tang Rui <rtang@vmware.com>
Date:   Sun Mar 18 19:59:30 2012 -0700

    Merge branch 'services-r9' into 'services-10'

    Change-Id: I81650bc1a08ac431aeceb97b686df17486f78c4d

commit f69403b
Author: Cliff Wang <cliffwang@vmware.com>
Date:   Sat Mar 17 18:01:52 2012 +0800

    Fix max connection test failure on some machines

    Change-Id: I37219c4f02993a03ae4d344b072761558630f3c1

commit 68b0d4f
Author: felixhoo <hhu@vmware.com>
Date:   Thu Mar 15 00:54:53 2012 -0600

    Update varz only when nats is enabled.

    Change-Id: I508a6bb8a795030ac7589857120130f0b5451e8c

commit ce1262b
Author: Yang Yang <yangy@vmware.com>
Date:   Mon Mar 12 13:29:39 2012 +0800

    added the feature to auto generate spec report for ci

    1. added spec report generation feature for ci for:
          atmos, base, filesystem, mongodb, mysql, neo4j,
          postgresql, rabbit, redis, serialization_data_server,
          service_broker, vblob

    2. added a namespace called ci with task spec:
        for the new unit test with reports generated, do:
          bundle exec rake ci:spec
        the reports would sit in spec_report in the xml format, which can be recognized by jenkins.

    Change-Id: Ic2ff8dd863dbb6bfe632929f8e9b556bb27bb53b

commit 5cab0f4
Merge: d87fd5a 7df1576
Author: Cliff Wang <cliffwang@vmware.com>
Date:   Tue Mar 13 09:06:25 2012 +0000

    Merge "make mongodb unit test pass even without dev_setup env" into services-r10

commit d87fd5a
Merge: f560734 7181377
Author: Yang Yang <yangy@vmware.com>
Date:   Tue Mar 13 07:07:27 2012 +0000

    Merge "fixed a bug in the service_broker unit test" into services-r10

commit 7181377
Author: Yang Yang <yangy@vmware.com>
Date:   Mon Mar 12 20:19:07 2012 +0800

    fixed a bug in the service_broker unit test

    case: when the directory for local_db does not exist, the unit test will return a failure
    fix: added code to check whether the directory for local_db exists or not, create a directory
         if it does not exist.

    Change-Id: Iacc0bdae3c564847e3793f9e3f636889a02e2683

commit 7df1576
Author: Cliff Wang <cliffwang@vmware.com>
Date:   Mon Mar 12 23:39:43 2012 -0600

    make mongodb unit test pass even without dev_setup env

    Change-Id: I4a722cbdbb79883bc4c82613cc7d7ffe936e37ad

commit f560734
Merge: 250def1 5ee7655
Author: figo <figof@mozy.com>
Date:   Mon Mar 12 18:48:03 2012 -0700

    Merge branch 'services-r9' into services-r10

    Change-Id: Ifd3d6f9435204825b635ce9f301c05fadb19ed37

commit 250def1
Merge: d63882e cd74fa6
Author: figo <figof@mozy.com>
Date:   Sun Mar 11 23:35:02 2012 -0700

    Merge branch 'master' into services-r10

    Conflicts:
    	atmos/Gemfile.lock
    	base/Gemfile.lock
    	base/lib/base/provisioner.rb
    	base/lib/base/version.rb
    	filesystem/Gemfile.lock
    	mongodb/Gemfile.lock
    	mysql/Gemfile.lock
    	neo4j/Gemfile.lock
    	postgresql/Gemfile.lock
    	redis/Gemfile.lock
    	service_broker/Gemfile.lock

    Change-Id: Iedc2e2ba29f714a581f8b84d44a4f87239113592

commit d63882e
Author: figo <figof@mozy.com>
Date:   Tue Feb 28 08:16:15 2012 -0800

    Revert "Revert "Merge branch 'services-r8' into services-r9""

    This reverts commit 409cd99.

    Change-Id: Iacd317820455695b924217a55fded5659e289cfb

Change-Id: I9f335be5f0ca124285eebdc145ca9cdcb56ace68
  • Loading branch information...
commit 6f4ce7109a5f6207b598bb7c455affc9e04a6690 1 parent fb5fdf4
figo authored
Showing with 3,893 additions and 1,707 deletions.
  1. +2 −2 Rakefile
  2. +2 −0  atmos/Gemfile
  3. +18 −13 atmos/Gemfile.lock
  4. +10 −3 atmos/Rakefile
  5. +23 −15 atmos/spec/Rakefile
  6. BIN  atmos/vendor/cache/little-plugger-1.1.3.gem
  7. BIN  atmos/vendor/cache/logging-1.6.1.gem
  8. BIN  atmos/vendor/cache/resque-1.19.0.gem
  9. BIN  atmos/vendor/cache/resque-1.20.0.gem
  10. BIN  atmos/vendor/cache/resque-status-0.2.4.gem
  11. BIN  atmos/vendor/cache/resque-status-0.3.2.gem
  12. BIN  atmos/vendor/cache/simplecov-0.5.4.gem
  13. BIN  atmos/vendor/cache/simplecov-html-0.5.3.gem
  14. BIN  atmos/vendor/cache/simplecov-rcov-0.2.3.gem
  15. BIN  atmos/vendor/cache/systemu-2.4.2.gem
  16. BIN  atmos/vendor/cache/systemu-2.5.0.gem
  17. BIN  atmos/vendor/cache/uuid-2.3.4.gem
  18. BIN  atmos/vendor/cache/uuid-2.3.5.gem
  19. BIN  atmos/vendor/cache/vcap_services_base-0.1.8.gem
  20. BIN  atmos/vendor/cache/vcap_services_base-0.1.9.gem
  21. BIN  atmos/vendor/cache/vegas-0.1.11.gem
  22. BIN  atmos/vendor/cache/vegas-0.1.8.gem
  23. +2 −0  base/Gemfile
  24. +23 −18 base/Gemfile.lock
  25. +10 −3 base/Rakefile
  26. +41 −0 base/lib/base/asynchronous_service_gateway.rb
  27. +5 −5 base/lib/base/backup.rb
  28. +1 −1  base/lib/base/base.rb
  29. +5 −0 base/lib/base/base_job.rb
  30. +12 −2 base/lib/base/gateway.rb
  31. +17 −37 base/lib/base/job/async_job.rb
  32. +27 −0 base/lib/base/job/config.rb
  33. +131 −0 base/lib/base/job/lock.rb
  34. +204 −74 base/lib/base/job/serialization.rb
  35. +205 −97 base/lib/base/job/snapshot.rb
  36. +110 −36 base/lib/base/node.rb
  37. +0 −2  base/lib/base/node_bin.rb
  38. +118 −8 base/lib/base/provisioner.rb
  39. +5 −0 base/lib/base/service_error.rb
  40. +1 −1  base/lib/base/version.rb
  41. +75 −0 base/lib/base/worker_bin.rb
  42. +2 −3 base/lib/vcap_services_base.rb
  43. +24 −16 base/spec/Rakefile
  44. +48 −0 base/spec/async_gw_spec.rb
  45. +50 −2 base/spec/helper/async_gw_spec_helper.rb
  46. +6 −0 base/spec/helper/job_spec_helper.rb
  47. +129 −2 base/spec/helper/node_spec_helper.rb
  48. +53 −1 base/spec/helper/provision_spec_helper.rb
  49. +105 −0 base/spec/job_spec.rb
  50. +147 −0 base/spec/node_spec.rb
  51. +63 −0 base/spec/provision_spec.rb
  52. +2 −2 base/vcap_services_base.gemspec
  53. +2 −0  filesystem/Gemfile
  54. +18 −13 filesystem/Gemfile.lock
  55. +10 −3 filesystem/Rakefile
  56. +24 −18 filesystem/spec/Rakefile
  57. BIN  filesystem/vendor/cache/little-plugger-1.1.3.gem
  58. BIN  filesystem/vendor/cache/logging-1.6.1.gem
  59. BIN  filesystem/vendor/cache/resque-1.19.0.gem
  60. BIN  filesystem/vendor/cache/resque-1.20.0.gem
  61. BIN  filesystem/vendor/cache/resque-status-0.2.4.gem
  62. BIN  filesystem/vendor/cache/resque-status-0.3.2.gem
  63. BIN  filesystem/vendor/cache/simplecov-0.5.4.gem
  64. BIN  filesystem/vendor/cache/simplecov-html-0.5.3.gem
  65. BIN  filesystem/vendor/cache/simplecov-rcov-0.2.3.gem
  66. BIN  filesystem/vendor/cache/systemu-2.4.2.gem
  67. BIN  filesystem/vendor/cache/systemu-2.5.0.gem
  68. BIN  filesystem/vendor/cache/uuid-2.3.4.gem
  69. BIN  filesystem/vendor/cache/uuid-2.3.5.gem
  70. BIN  filesystem/vendor/cache/vcap_services_base-0.1.8.gem
  71. BIN  filesystem/vendor/cache/vcap_services_base-0.1.9.gem
  72. BIN  filesystem/vendor/cache/vegas-0.1.11.gem
  73. BIN  filesystem/vendor/cache/vegas-0.1.8.gem
  74. +2 −0  mongodb/Gemfile
  75. +18 −13 mongodb/Gemfile.lock
  76. +8 −27 mongodb/Rakefile
  77. +18 −5 mongodb/bin/mongodb_worker
  78. +1 −2  mongodb/config/mongodb_worker.yml
  79. +15 −89 mongodb/lib/mongodb_service/job/mongodb_serialization.rb
  80. +8 −33 mongodb/lib/mongodb_service/job/mongodb_snapshot.rb
  81. +17 −3 mongodb/lib/mongodb_service/mongodb_node.rb
  82. +10 −29 mongodb/lib/mongodb_service/mongodb_provisioner.rb
  83. +24 −16 mongodb/spec/Rakefile
  84. +1 −1  mongodb/spec/config/mongodb_backup.yml.erb
  85. +2 −1  mongodb/spec/mongodb_backup_spec.rb
  86. +18 −5 mongodb/spec/mongodb_provision_spec.rb
  87. +16 −2 mongodb/spec/mongodb_rebalance_spec.rb
  88. BIN  mongodb/vendor/cache/little-plugger-1.1.3.gem
  89. BIN  mongodb/vendor/cache/logging-1.6.1.gem
  90. BIN  mongodb/vendor/cache/resque-1.19.0.gem
  91. BIN  mongodb/vendor/cache/resque-1.20.0.gem
  92. BIN  mongodb/vendor/cache/resque-status-0.2.4.gem
  93. BIN  mongodb/vendor/cache/resque-status-0.3.2.gem
  94. BIN  mongodb/vendor/cache/simplecov-0.5.4.gem
  95. BIN  mongodb/vendor/cache/simplecov-html-0.5.3.gem
  96. BIN  mongodb/vendor/cache/simplecov-rcov-0.2.3.gem
  97. BIN  mongodb/vendor/cache/systemu-2.4.2.gem
  98. BIN  mongodb/vendor/cache/systemu-2.5.0.gem
  99. BIN  mongodb/vendor/cache/uuid-2.3.4.gem
  100. BIN  mongodb/vendor/cache/uuid-2.3.5.gem
  101. BIN  mongodb/vendor/cache/vcap_services_base-0.1.8.gem
  102. BIN  mongodb/vendor/cache/vcap_services_base-0.1.9.gem
  103. BIN  mongodb/vendor/cache/vegas-0.1.11.gem
  104. BIN  mongodb/vendor/cache/vegas-0.1.8.gem
  105. +2 −3 mysql/Gemfile
  106. +18 −16 mysql/Gemfile.lock
  107. +8 −27 mysql/Rakefile
  108. +2 −2 mysql/bin/mysql_backup
  109. +18 −5 mysql/bin/mysql_worker
  110. +1 −2  mysql/config/mysql_worker.yml
  111. +13 −76 mysql/lib/mysql_service/job/mysql_serialization.rb
  112. +9 −37 mysql/lib/mysql_service/job/mysql_snapshot.rb
  113. +19 −4 mysql/lib/mysql_service/node.rb
  114. +10 −29 mysql/lib/mysql_service/provisioner.rb
  115. +24 −16 mysql/spec/Rakefile
  116. +22 −1 mysql/spec/mysql_node_spec.rb
  117. BIN  mysql/vendor/cache/little-plugger-1.1.3.gem
  118. BIN  mysql/vendor/cache/logging-1.6.1.gem
  119. BIN  mysql/vendor/cache/resque-1.19.0.gem
  120. BIN  mysql/vendor/cache/resque-1.20.0.gem
  121. BIN  mysql/vendor/cache/resque-status-0.2.4.gem
  122. BIN  mysql/vendor/cache/resque-status-0.3.2.gem
  123. BIN  mysql/vendor/cache/simplecov-0.5.4.gem
  124. BIN  mysql/vendor/cache/simplecov-html-0.5.3.gem
  125. BIN  mysql/vendor/cache/simplecov-rcov-0.2.3.gem
  126. BIN  mysql/vendor/cache/systemu-2.4.1.gem
  127. BIN  mysql/vendor/cache/systemu-2.5.0.gem
  128. BIN  mysql/vendor/cache/uuid-2.3.4.gem
  129. BIN  mysql/vendor/cache/uuid-2.3.5.gem
  130. BIN  mysql/vendor/cache/vcap_services_base-0.1.8.gem
  131. BIN  mysql/vendor/cache/vcap_services_base-0.1.9.gem
  132. BIN  mysql/vendor/cache/vegas-0.1.11.gem
  133. BIN  mysql/vendor/cache/vegas-0.1.8.gem
  134. +2 −0  neo4j/Gemfile
  135. +18 −13 neo4j/Gemfile.lock
  136. +10 −3 neo4j/Rakefile
  137. +1 −1  neo4j/lib/neo4j_service/neo4j_node.rb
  138. +24 −16 neo4j/spec/Rakefile
  139. +3 −2 neo4j/spec/neo4j_node_bind_spec.rb
  140. +3 −2 neo4j/spec/neo4j_node_provision_spec.rb
  141. BIN  neo4j/vendor/cache/little-plugger-1.1.3.gem
  142. BIN  neo4j/vendor/cache/logging-1.6.1.gem
  143. BIN  neo4j/vendor/cache/resque-1.19.0.gem
  144. BIN  neo4j/vendor/cache/resque-1.20.0.gem
  145. BIN  neo4j/vendor/cache/resque-status-0.2.4.gem
  146. BIN  neo4j/vendor/cache/resque-status-0.3.2.gem
  147. BIN  neo4j/vendor/cache/simplecov-0.5.4.gem
  148. BIN  neo4j/vendor/cache/simplecov-html-0.5.3.gem
  149. BIN  neo4j/vendor/cache/simplecov-rcov-0.2.3.gem
  150. BIN  neo4j/vendor/cache/systemu-2.4.2.gem
  151. BIN  neo4j/vendor/cache/systemu-2.5.0.gem
  152. BIN  neo4j/vendor/cache/uuid-2.3.4.gem
  153. BIN  neo4j/vendor/cache/uuid-2.3.5.gem
  154. BIN  neo4j/vendor/cache/vcap_services_base-0.1.8.gem
  155. BIN  neo4j/vendor/cache/vcap_services_base-0.1.9.gem
  156. BIN  neo4j/vendor/cache/vegas-0.1.11.gem
  157. BIN  neo4j/vendor/cache/vegas-0.1.8.gem
  158. +2 −1  postgresql/Gemfile
  159. +18 −13 postgresql/Gemfile.lock
  160. +12 −3 postgresql/Rakefile
  161. +44 −7 postgresql/bin/postgresql_backup
  162. +9 −1 postgresql/bin/postgresql_gateway
  163. +1 −0  postgresql/bin/postgresql_node
  164. +19 −0 postgresql/bin/postgresql_worker
  165. +1 −0  postgresql/config/postgresql_backup.yml
  166. +10 −0 postgresql/config/postgresql_gateway.yml
  167. +2 −0  postgresql/config/postgresql_node.yml
  168. +25 −0 postgresql/config/postgresql_worker.yml
  169. +17 −0 postgresql/lib/postgresql_service/job.rb
  170. +134 −0 postgresql/lib/postgresql_service/job/postgresql_serialization.rb
  171. +100 −0 postgresql/lib/postgresql_service/job/postgresql_snapshot.rb
  172. +33 −0 postgresql/lib/postgresql_service/model.rb
  173. +90 −93 postgresql/lib/postgresql_service/node.rb
  174. +1 −0  postgresql/lib/postgresql_service/postgresql_error.rb
  175. +33 −1 postgresql/lib/postgresql_service/provisioner.rb
  176. +165 −1 postgresql/lib/postgresql_service/util.rb
  177. +24 −16 postgresql/spec/Rakefile
  178. +106 −18 postgresql/spec/postgresql_node_spec.rb
  179. +2 −1  postgresql/spec/spec_helper.rb
  180. BIN  postgresql/vendor/cache/little-plugger-1.1.3.gem
  181. BIN  postgresql/vendor/cache/logging-1.6.1.gem
  182. BIN  postgresql/vendor/cache/resque-1.19.0.gem
  183. BIN  postgresql/vendor/cache/resque-1.20.0.gem
  184. BIN  postgresql/vendor/cache/resque-status-0.2.4.gem
  185. BIN  postgresql/vendor/cache/resque-status-0.3.2.gem
  186. BIN  postgresql/vendor/cache/simplecov-0.5.4.gem
  187. BIN  postgresql/vendor/cache/simplecov-html-0.5.3.gem
  188. BIN  postgresql/vendor/cache/simplecov-rcov-0.2.3.gem
  189. BIN  postgresql/vendor/cache/systemu-2.4.2.gem
  190. BIN  postgresql/vendor/cache/systemu-2.5.0.gem
  191. BIN  postgresql/vendor/cache/uuid-2.3.4.gem
  192. BIN  postgresql/vendor/cache/uuid-2.3.5.gem
  193. BIN  postgresql/vendor/cache/vcap_services_base-0.1.8.gem
  194. BIN  postgresql/vendor/cache/vcap_services_base-0.1.9.gem
  195. BIN  postgresql/vendor/cache/vegas-0.1.11.gem
  196. BIN  postgresql/vendor/cache/vegas-0.1.8.gem
  197. +2 −1  rabbit/Gemfile
  198. +18 −13 rabbit/Gemfile.lock
  199. +10 −3 rabbit/Rakefile
  200. +0 −1  rabbit/bin/rabbit_node
  201. +2 −0  rabbit/lib/rabbit_service/rabbit_node.rb
  202. +25 −18 rabbit/spec/Rakefile
  203. +22 −3 rabbit/spec/node_spec.rb
  204. BIN  rabbit/vendor/cache/little-plugger-1.1.3.gem
  205. BIN  rabbit/vendor/cache/logging-1.6.1.gem
  206. BIN  rabbit/vendor/cache/resque-1.19.0.gem
  207. BIN  rabbit/vendor/cache/resque-1.20.0.gem
  208. BIN  rabbit/vendor/cache/resque-status-0.2.4.gem
  209. BIN  rabbit/vendor/cache/resque-status-0.3.2.gem
  210. BIN  rabbit/vendor/cache/simplecov-0.5.4.gem
  211. BIN  rabbit/vendor/cache/simplecov-html-0.5.3.gem
  212. BIN  rabbit/vendor/cache/simplecov-rcov-0.2.3.gem
  213. BIN  rabbit/vendor/cache/systemu-2.4.2.gem
  214. BIN  rabbit/vendor/cache/systemu-2.5.0.gem
  215. BIN  rabbit/vendor/cache/uuid-2.3.4.gem
  216. BIN  rabbit/vendor/cache/uuid-2.3.5.gem
  217. BIN  rabbit/vendor/cache/vcap_services_base-0.1.8.gem
  218. BIN  rabbit/vendor/cache/vcap_services_base-0.1.9.gem
  219. BIN  rabbit/vendor/cache/vegas-0.1.11.gem
  220. BIN  rabbit/vendor/cache/vegas-0.1.8.gem
  221. +2 −1  redis/Gemfile
  222. +18 −14 redis/Gemfile.lock
  223. +8 −27 redis/Rakefile
  224. +1 −0  redis/bin/redis_node
  225. +18 −5 redis/bin/redis_worker
  226. +1 −0  redis/config/redis_node.yml
  227. +1 −2  redis/config/redis_worker.yml
  228. +20 −87 redis/lib/redis_service/job/redis_serialization.rb
  229. +11 −36 redis/lib/redis_service/job/redis_snapshot.rb
  230. +61 −19 redis/lib/redis_service/redis_node.rb
  231. +10 −30 redis/lib/redis_service/redis_provisioner.rb
  232. +28 −12 redis/lib/redis_service/util.rb
  233. +25 −18 redis/spec/Rakefile
  234. +32 −7 redis/spec/node_spec.rb
  235. BIN  redis/vendor/cache/little-plugger-1.1.3.gem
  236. BIN  redis/vendor/cache/logging-1.6.1.gem
  237. BIN  redis/vendor/cache/resque-1.19.0.gem
  238. BIN  redis/vendor/cache/resque-1.20.0.gem
  239. BIN  redis/vendor/cache/resque-status-0.2.4.gem
  240. BIN  redis/vendor/cache/resque-status-0.3.2.gem
  241. BIN  redis/vendor/cache/simplecov-0.5.4.gem
  242. BIN  redis/vendor/cache/simplecov-html-0.5.3.gem
  243. BIN  redis/vendor/cache/simplecov-rcov-0.2.3.gem
  244. BIN  redis/vendor/cache/systemu-2.5.0.gem
  245. BIN  redis/vendor/cache/uuid-2.3.4.gem
  246. BIN  redis/vendor/cache/uuid-2.3.5.gem
  247. BIN  redis/vendor/cache/vcap_services_base-0.1.8.gem
  248. BIN  redis/vendor/cache/vcap_services_base-0.1.9.gem
  249. BIN  redis/vendor/cache/vegas-0.1.11.gem
  250. BIN  redis/vendor/cache/vegas-0.1.8.gem
  251. +3 −2 serialization_data_server/Gemfile
  252. +13 −4 serialization_data_server/Gemfile.lock
  253. +10 −4 serialization_data_server/Rakefile
  254. +0 −2  serialization_data_server/bin/serialization_data_server
  255. +47 −0 serialization_data_server/lib/server.rb
  256. +24 −16 serialization_data_server/spec/Rakefile
  257. +2 −2 serialization_data_server/spec/server_spec.rb
  258. BIN  serialization_data_server/vendor/cache/ffi-1.0.11.gem
  259. BIN  serialization_data_server/vendor/cache/little-plugger-1.1.3.gem
  260. BIN  serialization_data_server/vendor/cache/logging-1.6.1.gem
  261. BIN  serialization_data_server/vendor/cache/multi_json-1.0.4.gem
  262. BIN  serialization_data_server/vendor/cache/simplecov-0.5.4.gem
  263. BIN  serialization_data_server/vendor/cache/simplecov-html-0.5.3.gem
  264. BIN  serialization_data_server/vendor/cache/simplecov-rcov-0.2.3.gem
  265. BIN  serialization_data_server/vendor/cache/sys-filesystem-1.0.0.gem
  266. +2 −1  service_broker/Gemfile
  267. +18 −13 service_broker/Gemfile.lock
  268. +10 −3 service_broker/Rakefile
  269. +24 −16 service_broker/spec/Rakefile
  270. +5 −0 service_broker/spec/service_broker_spec.rb
  271. BIN  service_broker/vendor/cache/little-plugger-1.1.3.gem
  272. BIN  service_broker/vendor/cache/logging-1.6.1.gem
  273. BIN  service_broker/vendor/cache/resque-1.19.0.gem
  274. BIN  service_broker/vendor/cache/resque-1.20.0.gem
  275. BIN  service_broker/vendor/cache/resque-status-0.2.4.gem
  276. BIN  service_broker/vendor/cache/resque-status-0.3.2.gem
  277. BIN  service_broker/vendor/cache/simplecov-0.5.4.gem
  278. BIN  service_broker/vendor/cache/simplecov-html-0.5.3.gem
  279. BIN  service_broker/vendor/cache/simplecov-rcov-0.2.3.gem
  280. BIN  service_broker/vendor/cache/systemu-2.4.2.gem
  281. BIN  service_broker/vendor/cache/systemu-2.5.0.gem
  282. BIN  service_broker/vendor/cache/uuid-2.3.4.gem
  283. BIN  service_broker/vendor/cache/uuid-2.3.5.gem
  284. BIN  service_broker/vendor/cache/vcap_services_base-0.1.8.gem
  285. BIN  service_broker/vendor/cache/vcap_services_base-0.1.9.gem
  286. BIN  service_broker/vendor/cache/vegas-0.1.11.gem
  287. BIN  service_broker/vendor/cache/vegas-0.1.8.gem
  288. +7 −2 tools/backup/manager/Gemfile
  289. +109 −16 tools/backup/manager/Gemfile.lock
  290. +17 −4 tools/backup/manager/Rakefile
  291. +27 −32 tools/backup/manager/bin/backup_manager
  292. +2 −0  tools/backup/manager/config/backup_manager.yml
  293. +67 −39 tools/backup/manager/lib/backup_manager/manager.rb
  294. +42 −58 tools/backup/manager/lib/backup_manager/rotator.rb
  295. +26 −14 tools/backup/manager/spec/Rakefile
  296. +160 −139 tools/backup/manager/spec/backup_manager_spec.rb
  297. +20 −30 tools/backup/manager/spec/spec_helper.rb
  298. BIN  tools/backup/manager/vendor/cache/addressable-2.2.4.gem
  299. BIN  tools/backup/manager/vendor/cache/addressable-2.2.6.gem
  300. BIN  tools/backup/manager/vendor/cache/bcrypt-ruby-2.1.4.gem
Sorry, we could not display the entire diff because too many files (382) changed.
View
4 Rakefile
@@ -1,4 +1,4 @@
-SERVICES_DIR = %w(atmos filesystem mongodb mysql neo4j postgresql rabbit redis service_broker vblob)
+SERVICES_DIR = %w(atmos filesystem mongodb mysql neo4j postgresql rabbit redis service_broker vblob tools/backup/manager)
desc "Run integration tests."
task "tests" do |t|
@@ -8,7 +8,7 @@ end
namespace "bundler" do
desc "Update base gem"
task "update_base" do
- system "cd base && rake bundler:install"
+ system "cd base && rm -rf pkg && rake bundler:install"
SERVICES_DIR.each do |dir|
puts ">>>>>>>> enter #{dir}"
system "rm -f #{dir}/vendor/cache/vcap_services_base-*.gem && cp base/pkg/vcap_services_base-*.gem #{dir}/vendor/cache && cd #{dir} && bundle install --local"
View
2  atmos/Gemfile
@@ -19,5 +19,7 @@ group :test do
gem "rake"
gem "rspec"
gem "rcov"
+ gem "simplecov"
+ gem "simplecov-rcov"
gem "ci_reporter"
end
View
31 atmos/Gemfile.lock
@@ -64,9 +64,6 @@ GEM
fastercsv (1.5.4)
json (1.4.6)
json_pure (1.6.5)
- little-plugger (1.1.3)
- logging (1.6.1)
- little-plugger (>= 1.1.2)
macaddr (1.5.0)
systemu (>= 2.4.0)
multi_json (1.0.4)
@@ -85,15 +82,15 @@ GEM
redisk (0.2.2)
redis (>= 0.1.1)
redis-namespace (>= 0.1.0)
- resque (1.19.0)
+ resque (1.20.0)
multi_json (~> 1.0)
redis-namespace (~> 1.0.2)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
- resque-status (0.2.4)
+ resque-status (0.3.2)
redisk (>= 0.2.1)
- resque (>= 1.3.1)
- uuid (>= 2.0.2)
+ resque (~> 1.19)
+ uuid (~> 2.3)
rspec (2.5.0)
rspec-core (~> 2.5.0)
rspec-expectations (~> 2.5.0)
@@ -103,17 +100,23 @@ GEM
diff-lcs (~> 1.1.2)
rspec-mocks (2.5.0)
ruby-hmac (0.4.0)
+ simplecov (0.5.4)
+ multi_json (~> 1.0.3)
+ simplecov-html (~> 0.5.3)
+ simplecov-html (0.5.3)
+ simplecov-rcov (0.2.3)
+ simplecov (>= 0.4.1)
sinatra (1.2.7)
rack (~> 1.1)
tilt (>= 1.2.2, < 2.0)
stringex (1.2.1)
- systemu (2.4.2)
+ systemu (2.5.0)
thin (1.3.1)
daemons (>= 1.0.9)
eventmachine (>= 0.12.6)
rack (>= 1.0.0)
tilt (1.3.3)
- uuid (2.3.4)
+ uuid (2.3.5)
macaddr (~> 1.0)
uuidtools (2.1.2)
vcap_common (1.0.8)
@@ -123,7 +126,7 @@ GEM
thin (~> 1.3.1)
yajl-ruby (~> 0.8.3)
vcap_logging (0.1.3)
- vcap_services_base (0.1.8)
+ vcap_services_base (0.1.9)
curb (~> 0.7.16)
datamapper (~> 1.1.0)
do_sqlite3 (~> 0.10.3)
@@ -131,16 +134,16 @@ GEM
eventmachine (~> 0.12.11.cloudfoundry.3)
eventmachine_httpserver (~> 0.2.1)
json (~> 1.4.6)
- logging (>= 1.5.0)
nats (~> 0.4.22.beta.8)
- resque-status (~> 0.2.4)
+ resque (~> 1.20)
+ resque-status (~> 0.3.2)
ruby-hmac (~> 0.4.0)
sinatra (~> 1.2.3)
thin (~> 1.3.1)
uuidtools (~> 2.1.2)
vcap_common (>= 1.0.8)
vcap_logging (>= 0.1.3)
- vegas (0.1.8)
+ vegas (0.1.11)
rack (>= 1.0.0)
xml-simple (1.0.15)
yajl-ruby (0.8.3)
@@ -159,6 +162,8 @@ DEPENDENCIES
rcov
rspec
ruby-hmac
+ simplecov
+ simplecov-rcov
sinatra
thin
uuidtools
View
13 atmos/Rakefile
@@ -3,9 +3,12 @@ require 'rake'
desc "Run specs"
task "spec" => ["bundler:install:test", "test:spec"]
-desc "Run specs using RCov"
+desc "Run specs using SimpleCov"
task "spec:rcov" => ["bundler:install:test", "test:spec:rcov"]
+desc "Run ci using SimpleCov"
+task "spec:ci" => ["bundler:install:test", "test:spec:ci"]
+
namespace "bundler" do
desc "Install gems"
task "install" do
@@ -33,7 +36,11 @@ namespace "test" do
sh("cd spec && ../../base/bin/nats-util start && rake spec && ../../base/bin/nats-util stop")
end
- task "spec:rcov" do |t|
- sh("cd spec && rake spec:rcov")
+ task "spec:rcov" do |t|
+ sh("cd spec && ../../base/bin/nats-util start && rake simcov && ../../base/bin/nats-util stop")
+ end
+
+ task "spec:ci" do |t|
+ sh("cd spec && ../../base/bin/nats-util start && rake spec:ci && ../../base/bin/nats-util stop")
end
end
View
38 atmos/spec/Rakefile
@@ -9,33 +9,41 @@ require 'rspec'
require 'rspec/core/rake_task'
require 'ci/reporter/rake/rspec'
+require "simplecov"
+require "simplecov-rcov"
+
coverage_dir = File.expand_path(File.join(File.dirname(__FILE__), "..", "spec_coverage"))
reports_dir = File.expand_path(File.join(File.dirname(__FILE__), "..", "spec_reports"))
-dump_file = File.join(Dir.tmpdir, "atmos.rcov")
-ignore_pattern = 'spec,[.]bundle,[/]gems[/]'
ENV['CI_REPORTS'] = reports_dir
desc "Run specs using RCov"
-task "spec:rcov" => ["ci:setup:rspec", "spec:rcov_internal", "convert_rcov_to_clover"]
+task "spec:ci" => ["ci:setup:rspec", "simcov"]
RSpec::Core::RakeTask.new do |t|
t.pattern = "**/*_spec.rb"
t.rspec_opts = ["--format", "documentation", "--colour"]
end
-desc "Run specs using RCov (internal, use spec:rcov instead)"
-RSpec::Core::RakeTask.new("spec:rcov_internal") do |t|
- sh("rm -f #{dump_file}")
- t.pattern = "**/*_spec.rb"
- t.rspec_opts = ["--format", "progress", "--colour"]
- t.rcov = true
- t.rcov_opts = ['--aggregate', dump_file, '--exclude', ignore_pattern, '--output', coverage_dir]
+desc "Run spec with coverage"
+task "simcov" => "cleanup_coverage" do
+ class SimpleCov::Formatter::CombinedFormatter
+ def format(result)
+ SimpleCov::Formatter::RcovFormatter.new.format(result)
+ end
+ end
+
+ SimpleCov.formatter = SimpleCov::Formatter::CombinedFormatter
+ SimpleCov.root('..')
+ SimpleCov.coverage_dir('spec_coverage')
+ SimpleCov.start do
+ add_filter "/spec/"
+ spec_dir = File.expand_path("..", __FILE__)
+ Rspec::Core::Runner.disable_autorun!
+ Rspec::Core::Runner.run([spec_dir], STDERR, STDOUT)
+ end
end
-task "convert_rcov_to_clover" do |t|
- analyzer = File.expand_path(File.join(File.dirname(__FILE__), "..", "..", "..", "tests", "common", "rcov_analyzer.rb"))
- clover_output = File.join(coverage_dir, "clover.xml")
- sh("ruby #{analyzer} #{dump_file} #{ignore_pattern} > #{clover_output}")
- sh("rm -f #{dump_file}")
+task "cleanup_coverage" do
+ rm_rf coverage_dir
end
View
BIN  atmos/vendor/cache/little-plugger-1.1.3.gem
Binary file not shown
View
BIN  atmos/vendor/cache/logging-1.6.1.gem
Binary file not shown
View
BIN  atmos/vendor/cache/resque-1.19.0.gem
Binary file not shown
View
BIN  atmos/vendor/cache/resque-1.20.0.gem
Binary file not shown
View
BIN  atmos/vendor/cache/resque-status-0.2.4.gem
Binary file not shown
View
BIN  atmos/vendor/cache/resque-status-0.3.2.gem
Binary file not shown
View
BIN  atmos/vendor/cache/simplecov-0.5.4.gem
Binary file not shown
View
BIN  atmos/vendor/cache/simplecov-html-0.5.3.gem
Binary file not shown
View
BIN  atmos/vendor/cache/simplecov-rcov-0.2.3.gem
Binary file not shown
View
BIN  atmos/vendor/cache/systemu-2.4.2.gem
Binary file not shown
View
BIN  atmos/vendor/cache/systemu-2.5.0.gem
Binary file not shown
View
BIN  atmos/vendor/cache/uuid-2.3.4.gem
Binary file not shown
View
BIN  atmos/vendor/cache/uuid-2.3.5.gem
Binary file not shown
View
BIN  atmos/vendor/cache/vcap_services_base-0.1.8.gem
Binary file not shown
View
BIN  atmos/vendor/cache/vcap_services_base-0.1.9.gem
Binary file not shown
View
BIN  atmos/vendor/cache/vegas-0.1.11.gem
Binary file not shown
View
BIN  atmos/vendor/cache/vegas-0.1.8.gem
Binary file not shown
View
2  base/Gemfile
@@ -8,4 +8,6 @@ group :test do
gem "rspec"
gem "rcov"
gem "ci_reporter"
+ gem "simplecov"
+ gem "simplecov-rcov"
end
View
41 base/Gemfile.lock
@@ -1,7 +1,7 @@
PATH
remote: .
specs:
- vcap_services_base (0.1.8)
+ vcap_services_base (0.1.9)
curb (~> 0.7.16)
datamapper (~> 1.1.0)
do_sqlite3 (~> 0.10.3)
@@ -9,9 +9,9 @@ PATH
eventmachine (~> 0.12.11.cloudfoundry.3)
eventmachine_httpserver (~> 0.2.1)
json (~> 1.4.6)
- logging (>= 1.5.0)
nats (~> 0.4.22.beta.8)
- resque-status (~> 0.2.4)
+ resque (~> 1.20)
+ resque-status (~> 0.3.2)
ruby-hmac (~> 0.4.0)
sinatra (~> 1.2.3)
thin (~> 1.3.1)
@@ -22,14 +22,14 @@ PATH
GEM
remote: http://rubygems.org/
specs:
- addressable (2.2.6)
+ addressable (2.2.7)
bcrypt-ruby (2.1.4)
builder (3.0.0)
ci_reporter (1.6.4)
builder (>= 2.1.2)
curb (0.7.16)
daemons (1.1.8)
- data_objects (0.10.7)
+ data_objects (0.10.8)
addressable (~> 2.1)
datamapper (1.1.0)
dm-aggregates (= 1.1.0)
@@ -67,8 +67,8 @@ GEM
uuidtools (~> 2.1.2)
dm-validations (1.1.0)
dm-core (~> 1.1.0)
- do_sqlite3 (0.10.7)
- data_objects (= 0.10.7)
+ do_sqlite3 (0.10.8)
+ data_objects (= 0.10.8)
em-http-request (0.3.0)
addressable (>= 2.0.0)
escape_utils
@@ -79,13 +79,10 @@ GEM
fastercsv (1.5.4)
json (1.4.6)
json_pure (1.6.5)
- little-plugger (1.1.3)
- logging (1.6.1)
- little-plugger (>= 1.1.2)
macaddr (1.5.0)
systemu (>= 2.4.0)
multi_json (1.0.4)
- nats (0.4.22.beta.8)
+ nats (0.4.22)
daemons (>= 1.1.4)
eventmachine (>= 0.12.10)
json_pure (>= 1.6.1)
@@ -100,15 +97,15 @@ GEM
redisk (0.2.2)
redis (>= 0.1.1)
redis-namespace (>= 0.1.0)
- resque (1.19.0)
+ resque (1.20.0)
multi_json (~> 1.0)
redis-namespace (~> 1.0.2)
sinatra (>= 0.9.2)
vegas (~> 0.1.2)
- resque-status (0.2.4)
+ resque-status (0.3.2)
redisk (>= 0.2.1)
- resque (>= 1.3.1)
- uuid (>= 2.0.2)
+ resque (~> 1.19)
+ uuid (~> 2.3)
rspec (2.5.0)
rspec-core (~> 2.5.0)
rspec-expectations (~> 2.5.0)
@@ -118,17 +115,23 @@ GEM
diff-lcs (~> 1.1.2)
rspec-mocks (2.5.0)
ruby-hmac (0.4.0)
+ simplecov (0.5.4)
+ multi_json (~> 1.0.3)
+ simplecov-html (~> 0.5.3)
+ simplecov-html (0.5.3)
+ simplecov-rcov (0.2.3)
+ simplecov (>= 0.4.1)
sinatra (1.2.3)
rack (~> 1.1)
tilt (>= 1.2.2, < 2.0)
stringex (1.2.2)
- systemu (2.4.2)
+ systemu (2.5.0)
thin (1.3.1)
daemons (>= 1.0.9)
eventmachine (>= 0.12.6)
rack (>= 1.0.0)
tilt (1.2.2)
- uuid (2.3.4)
+ uuid (2.3.5)
macaddr (~> 1.0)
uuidtools (2.1.2)
vcap_common (1.0.10)
@@ -139,7 +142,7 @@ GEM
yajl-ruby (~> 0.8.3)
vcap_logging (1.0.0)
rake
- vegas (0.1.8)
+ vegas (0.1.11)
rack (>= 1.0.0)
yajl-ruby (0.8.3)
@@ -151,5 +154,7 @@ DEPENDENCIES
rake
rcov
rspec
+ simplecov
+ simplecov-rcov
sinatra
vcap_services_base!
View
13 base/Rakefile
@@ -4,9 +4,12 @@ require 'bundler'
desc "Run specs"
task "spec" => ["bundler:install:test", "test:spec"]
-desc "Run specs using RCov"
+desc "Run specs using SimpleCov"
task "spec:rcov" => ["bundler:install:test", "test:spec:rcov"]
+desc "Run ci using SimpleCov"
+task "spec:ci" => ["bundler:install:test", "test:spec:ci"]
+
namespace "bundler" do
gem_helper = Bundler::GemHelper.new(Dir.pwd)
desc "Build gem package"
@@ -44,7 +47,11 @@ namespace "test" do
sh("cd spec && ../bin/nats-util start && rake spec && ../bin/nats-util stop")
end
- task "spec:rcov" do |t|
- sh("cd spec && rake spec:rcov")
+ task "spec:rcov" do |t|
+ sh("cd spec && ../bin/nats-util start && rake simcov && ../bin/nats-util stop")
+ end
+
+ task "spec:ci" do |t|
+ sh("cd spec && ../bin/nats-util start && rake spec:ci && ../bin/nats-util stop")
end
end
View
41 base/lib/base/asynchronous_service_gateway.rb
@@ -287,6 +287,22 @@ def check_orphan(handles, callback, errback)
async_mode
end
+ # Delete a snapshot
+ delete "/gateway/v1/configurations/:service_id/snapshots/:snapshot_id" do
+ not_impl unless @api_extensions.include? "snapshots"
+ service_id = params["service_id"]
+ snapshot_id = params["snapshot_id"]
+ @logger.info("Delete service_id=#{service_id} to snapshot_id=#{snapshot_id}")
+ @provisioner.delete_snapshot(service_id, snapshot_id) do |msg|
+ if msg['success']
+ async_reply(VCAP::Services::Api::Job.new(msg['response']).encode)
+ else
+ async_reply_error(msg['response'])
+ end
+ end
+ async_mode
+ end
+
# Get serialized url
get "/gateway/v1/configurations/:service_id/serialized/url" do
not_impl unless @api_extensions.include? "serialization"
@@ -412,6 +428,31 @@ def check_orphan(handles, callback, errback)
async_mode
end
+ # Service migration API
+ post "/service/internal/v1/migration/:node_id/:instance_id/:action" do
+ @logger.info("Migration: #{params["action"]} instance #{params["instance_id"]} in #{params["node_id"]}")
+ @provisioner.migrate_instance(params["node_id"], params["instance_id"], params["action"]) do |msg|
+ if msg["success"]
+ async_reply(msg["response"].to_json)
+ else
+ async_reply_error(msg["response"])
+ end
+ end
+ async_mode
+ end
+
+ get "/service/internal/v1/migration/:node_id/instances" do
+ @logger.info("Migration: get instance id list of node #{params["node_id"]}")
+ @provisioner.get_instance_id_list(params["node_id"]) do |msg|
+ if msg["success"]
+ async_reply(msg["response"].to_json)
+ else
+ async_reply_error(msg["response"])
+ end
+ end
+ async_mode
+ end
+
#################### Helpers ####################
View
10 base/lib/base/backup.rb
@@ -3,8 +3,6 @@
require 'optparse'
require 'timeout'
require 'fileutils'
-require 'logger'
-require 'logging'
require 'yaml'
require 'pathname'
@@ -110,13 +108,15 @@ def start
echo "Backup is interrupted!"
end
- def get_dump_path(name,mode=0)
+ def get_dump_path(name, options={})
name = name.sub(/^(mongodb|redis)-/,'')
+ mode = options[:mode] || 0
+ time = options[:time] || Time.now
case mode
when 1
- File.join(@config['backup_base_dir'], 'backups', @config['service_name'],name, Time.new.to_i.to_s,@config['node_id'])
+ File.join(@config['backup_base_dir'], 'backups', @config['service_name'],name, time.to_i.to_s,@config['node_id'])
else
- File.join(@config['backup_base_dir'], 'backups', @config['service_name'], name[0,2], name[2,2], name[4,2], name, Time.new.to_i.to_s)
+ File.join(@config['backup_base_dir'], 'backups', @config['service_name'], name[0,2], name[2,2], name[4,2], name, time.to_i.to_s)
end
end
View
2  base/lib/base/base.rb
@@ -74,7 +74,7 @@ def service_description()
def publish(reply, msg)
# nats publish are only allowed to be called in reactor thread.
EM.schedule do
- @node_nats.publish(reply, msg)
+ @node_nats.publish(reply, msg) if @node_nats
end
end
View
5 base/lib/base/base_job.rb
@@ -0,0 +1,5 @@
+# Copyright (c) 2009-2011 VMware, Inc.
+require 'job/config'
+require 'job/snapshot'
+require 'job/serialization'
+require 'job/lock'
View
14 base/lib/base/gateway.rb
@@ -3,8 +3,6 @@
require 'bundler/setup'
require 'optparse'
-require 'logger'
-require 'logging'
require 'net/http'
require 'thin'
require 'yaml'
@@ -15,6 +13,7 @@
$LOAD_PATH.unshift File.dirname(__FILE__)
require 'asynchronous_service_gateway'
+require 'job/config'
require 'abstract'
module VCAP
@@ -59,6 +58,15 @@ def setup_vcap_logging
@config[:logger] = logger
end
+ def setup_async_job_config
+ resque = @config[:resque]
+ if resque
+ resque = VCAP.symbolize_keys(resque)
+ VCAP::Services::Base::AsyncJob::Config.redis_config = resque
+ VCAP::Services::Base::AsyncJob::Config.logger = @config[:logger]
+ end
+ end
+
def setup_pid
if @config[:pid]
pf = VCAP::PidFile.new(@config[:pid])
@@ -73,6 +81,8 @@ def start
setup_pid
+ setup_async_job_config
+
@config[:host] = VCAP.local_ip(@config[:ip_route])
@config[:port] ||= VCAP.grab_ephemeral_port
@config[:service][:label] = "#{@config[:service][:name]}-#{@config[:service][:version]}"
View
54 base/lib/base/job/async_job.rb
@@ -1,6 +1,8 @@
# Copyright (c) 2009-2011 VMware, Inc.
-require "resque/job_with_status"
+require "resque-status"
+$LOAD_PATH.unshift File.dirname(__FILE__)
+require "config"
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..')
require "service_error"
@@ -13,56 +15,34 @@ def enqueue(klass, *args)
enqueue_to(queue, klass, *args)
end
- # Backport from resque master branch, we can remove this method when gem is updated.
- def enqueue_to(queue, klass, *args)
- # Perform before_enqueue hooks. Don't perform enqueue if any hook returns false
- before_hooks = Plugin.before_enqueue_hooks(klass).collect do |hook|
- klass.send(hook, *args)
- end
- return nil if before_hooks.any? { |result| result == false }
-
- Job.create(queue, klass, *args)
-
- Plugin.after_enqueue_hooks(klass).each do |hook|
- klass.send(hook, *args)
- end
-
- return true
- end
-
- class Status
- # new attributes
- hash_accessor :complete_time
- end
end
-module VCAP
- module Services
- end
+module Resque::Plugins::Status
+ class Hash
+ # new attributes
+ hash_accessor :complete_time
+ end
end
# A thin layer wraps resque-status
-module VCAP::Services::AsyncJob
+module VCAP::Services::Base::AsyncJob
include VCAP::Services::Base::Error
- def self.logger=(logger)
- @logger = logger
- end
-
- def job_repo_setup(options={})
- raise "AsyncJob requires redis configuration." unless options[:redis]
- @logger.debug("Initialize Resque using #{options}")
- Resque.redis = options[:redis]
- Resque::Status.expire_in = options[:expire] if options[:expire]
+ def job_repo_setup
+ redis = Config.redis
+ @logger = Config.logger
+ raise "AsyncJob requires redis configuration." unless redis
+ @logger.debug("Initialize Resque using #{redis}") if @logger
+ ::Resque.redis = redis
end
def get_job(jobid)
- res = Resque::Status.get(jobid)
+ res = Resque::Plugins::Status::Hash.get(jobid)
job_to_json(res)
end
def get_all_jobs()
- Resque::Status.status_ids
+ Resque::Plugins::Status::Hash.keys
end
def job_to_json(job)
View
27 base/lib/base/job/config.rb
@@ -0,0 +1,27 @@
+# Copyright (c) 2009-2012 VMware, Inc.
+require "redis"
+require "resque"
+
+module VCAP
+ module Services
+ module Base
+ module AsyncJob
+ end
+ end
+ end
+end
+
+class VCAP::Services::Base::AsyncJob::Config
+ class << self
+ attr_reader :redis_config, :redis, :logger
+ def redis_config=(config)
+ @redis_config = config
+ @redis = ::Redis.new config
+ Resque.redis = @redis
+ end
+
+ def logger=(logger)
+ @logger = logger
+ end
+ end
+end
View
131 base/lib/base/job/lock.rb
@@ -0,0 +1,131 @@
+# Copyright (c) 2009-2011 VMware, Inc.
+require "logger"
+require "redis"
+
+$LOAD_PATH.unshift File.dirname(__FILE__)
+require "config"
+
+# redis locking primitive using setnx.
+# http://redis.io/commands/setnx
+module VCAP::Services::Base::AsyncJob
+ class Lock
+ attr_reader :expiration, :timeout, :name
+ include VCAP::Services::Base::Error
+
+ def initialize(name, opts={})
+ @name = name
+ @timeout = opts[:timeout] || 10 #seconds
+ @expiration = opts[:expiration] || 10 # seconds
+ @logger = opts[:logger] || make_logger
+ config = Config.redis_config
+ raise "Can't find configuration of redis." unless config
+ @redis = ::Redis.new(config)
+ @released_thread = {}
+ end
+
+ def make_logger
+ logger = Logger.new(STDOUT)
+ logger.level = Logger::ERROR
+ logger
+ end
+
+ def lock
+ @logger.debug("Acquiring lock: #{@name}")
+ started = Time.now.to_f
+ expiration = started.to_f + @expiration + 1
+ until @redis.setnx(@name, expiration)
+ existing_lock = @redis.get(@name)
+ if existing_lock.to_f < Time.now.to_f
+ @logger.debug("Lock #{@name} is expired, trying to acquire it.")
+ break if watch_and_update(@redis, expiration)
+ end
+
+ raise ServiceError.new(ServiceError::JOB_QUEUE_TIMEOUT, @timeout)if Time.now.to_f - started > @timeout
+
+ sleep(1)
+
+ expiration = Time.now.to_f + @expiration + 1
+ end
+
+ @lock_expiration = expiration
+ refresh_thread = setup_refresh_thread
+ @logger.debug("Lock #{@name} is acquired, will expire at #{@lock_expiration}")
+
+ begin
+ yield if block_given?
+ ensure
+ release_thread(refresh_thread) if refresh_thread
+ delete
+ end
+ end
+
+ def watch_and_update(redis, expiration)
+ redis.watch(@name)
+ res = redis.multi do
+ redis.set(@name, expiration)
+ end
+ if res
+ @logger.debug("Lock #{@name} is renewed and acquired.")
+ else
+ @logger.debug("Lock #{@name} was updated by others.")
+ end
+ res
+ end
+
+ def release_thread t
+ @released_thread[t.object_id] = true
+ end
+
+ def released?
+ @released_thread[Thread.current.object_id]
+ end
+
+ def setup_refresh_thread
+ t = Thread.new do
+ redis = ::Redis.new(Config.redis_config)
+ sleep_interval = [1.0, @expiration/2].max
+ begin
+ loop do
+ break if released?
+ @logger.debug("Renewing lock #{@name}")
+ redis.watch(@name)
+ existing_lock = redis.get(@name)
+
+ break if existing_lock.to_f > @lock_expiration # lock has been updated by others
+ expiration = Time.now.to_f + @expiration + 1
+ break unless watch_and_update(redis, expiration)
+ @lock_expiration = expiration
+ sleep sleep_interval
+ end
+ rescue => e
+ @logger.error("Can't renew lock #{@name}, #{e}")
+ ensure
+ begin
+ @logger.debug("Lock renew thread for #{@name} exited.")
+ redis.quit
+ rescue => e
+ # just logging, ignore error
+ @logger.debug("Ignore error when quit: #{e}")
+ end
+ end
+ end
+ t
+ end
+
+ def delete
+ @logger.debug("Deleting lock: #{@name}")
+ existing_lock = @redis.get(@name)
+ @logger.debug("Lock #{@name} is acquired by others.")if existing_lock.to_f > @lock_expiration
+ @redis.watch(@name)
+ res = @redis.multi do
+ @redis.del(@name)
+ end
+ if res
+ @logger.debug("Lock #{@name} is deleted.")
+ else
+ @logger.debug("Lock #{@name} is acquired by others.")
+ end
+ true
+ end
+ end
+end
View
278 base/lib/base/job/serialization.rb
@@ -1,113 +1,243 @@
# Copyright (c) 2009-2011 VMware, Inc.
-require "resque/job_with_status"
+require "resque-status"
require "fileutils"
require "curb"
require "vcap/logging"
+# explictly import uuid to resolve namespace conflict between uuid and uuidtools gems.
+require "uuid"
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..')
require "base/service_error"
-module VCAP
- module Services
- end
-end
-
-module VCAP::Services::Serialization
- SERIALIZATION_KEY_PREFIX = "vcap:serialization".freeze
+module VCAP::Services::Base::AsyncJob
+ module Serialization
+ SERIALIZATION_KEY_PREFIX = "vcap:serialization".freeze
- class << self
- attr_reader :redis, :logger
+ class << self
+ attr_reader :redis
- # Config the redis using options
- def redis_connect(opts)
- resque = %w(host port password).inject({}){|res, o| res[o.to_sym] = opts[o]; res}
- @redis = ::Redis.new(resque)
+ def redis_connect
+ @redis = ::Redis.new(Config.redis_config)
+ end
end
- def redis=(redis)
- raise "Serialization requires redis configuration." unless redis
- @redis = redis
+ def redis_key(key)
+ "#{SERIALIZATION_KEY_PREFIX}:#{key}"
end
- def logger=(logger)
- @logger = logger
+ def fmt_error(e)
+ "#{e}: [#{e.backtrace.join(" | ")}]"
end
- end
- def redis_key(key)
- "#{SERIALIZATION_KEY_PREFIX}:#{key}"
- end
+ class SerializationJob
+ attr_reader :name
+ include Serialization
+ include Resque::Plugins::Status
+ include VCAP::Services::Base::Error
+
+ class << self
+
+ def queue_lookup_key
+ :node_id
+ end
+
+ def select_queue(*args)
+ result = nil
+ args.each do |arg|
+ result = arg[queue_lookup_key]if (arg.is_a? Hash )&& (arg.has_key?(queue_lookup_key))
+ end
+ @logger = Config.logger
+ @logger.info("Select queue #{result} for job #{self.class} with args:#{args.inspect}") if @logger
+ result
+ end
+ end
- class SerializationJob < Resque::JobWithStatus
- include VCAP::Services::Serialization
- include VCAP::Services::Base::Error
+ def initialize(*args)
+ super(*args)
+ parse_config
+ init_worker_logger
+ Serialization.redis_connect
+ end
- class << self
- attr_reader :logger
+ def create_lock
+ lock_name = "lock:lifecycle:#{name}"
+ lock = Lock.new(lock_name, :logger => @logger)
+ lock
+ end
+
+ def client
+ Serialization.redis
+ end
+
+ def init_worker_logger
+ @logger = Config.logger
+ end
+
+ def handle_error(e)
+ @logger.error("Error in #{self.class} uuid:#{@uuid}: #{fmt_error(e)}")
+ err = (e.instance_of?(ServiceError)? e : ServiceError.new(ServiceError::INTERNAL_ERROR)).to_hash
+ err_msg = Yajl::Encoder.encode(err)
+ failed(err_msg)
+ end
+
+ def required_options(*args)
+ missing_opts = args.select{|arg| !options.has_key? arg.to_s}
+ raise ArgumentError, "Missing #{missing_opts.join(', ')} in options: #{options.inspect}" unless missing_opts.empty?
+ end
- def queue_lookup_key
- :node_id
+ # the serialize path structure looks like <base-dir>\serialize\<service-name>\<aa>\<bb>\<cc>\
+ # <aabbcc-rest-of-instance-guid>\<serialization data>
+ def get_serialized_data_path(name)
+ File.join(@config["serialization_base_dir"], "serialize", @config["service_name"] , name[0,2],name[2,2], name[4,2], name)
end
- def logger=(logger)
- @logger = logger
+ # Update the download token for a serialized file and save it in redis
+ def update_download_token(service, name, file_name, token)
+ key = "#{service}:#{name}:token"
+ client.hset(redis_key(key), :token, token)
+ client.hset(redis_key(key), :file, file_name)
end
- def select_queue(*args)
- result = nil
- args.each do |arg|
- result = arg[queue_lookup_key]if (arg.is_a? Hash )&& (arg.has_key?(queue_lookup_key))
+ def parse_config
+ @config = Yajl::Parser.parse(ENV['WORKER_CONFIG'])
+ raise "Need environment variable: WORKER_CONFIG" unless @config
+ end
+
+ def cleanup(name)
+ return unless name
+ FileUtils.rm_rf(get_serialized_data_path(name))
+ end
+
+ # Validate the serialized data file.
+ # Sub class should override this method to supply specific validation.
+ def validate_input(file_path)
+ File.open(file_path) do |f|
+ return nil unless f.size > 0
end
- @logger.info("Select queue #{result} for job #{self.class} with args:#{args.inspect}") if @logger
- result
+ true
end
- end
- def initialize(*args)
- super(*args)
- parse_config
- init_worker_logger()
end
- def client
- VCAP::Services::Serialization.redis
- end
+ class BaseCreateSerializedURLJob < SerializationJob
+ attr_reader :dump_path
+
+ VALID_CREDENTIAL_CHARACTERS = ("A".."Z").to_a + ("a".."z").to_a + ("0".."9").to_a
+
+ # workflow template
+ # Sub class should return a hash contains filename that generated on shared storage. For example
+ # {:dump_file_name => 'db1.tgz'}
+ def perform
+ begin
+ required_options :service_id
+ @name = options["service_id"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ lock = create_lock
+ lock.lock do
+ result = execute
+ @logger.info("Results of create serialized url: #{result}")
+
+ token = generate_download_token()
+ service_name = @config["service_name"]
+ update_download_token(service_name, name, result[:dump_file_name], token)
+ url = generate_download_url(name, token)
+ @logger.info("Download link generated for #{name}: #{url}")
+
+ job_result = { :url => url }
+ completed(Yajl::Encoder.encode(job_result))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ cleanup(name)
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ end
+ end
- def init_worker_logger
- VCAP::Logging.setup_from_config(@config["logging"])
- @logger = VCAP::Logging.logger("#{@config["service_name"]}_worker")
- end
+ def generate_download_token(length=12)
+ Array.new(length) { VALID_CREDENTIAL_CHARACTERS[rand(VALID_CREDENTIAL_CHARACTERS.length)] }.join
+ end
- # the serialize path structure looks like <base-dir>\serialize\<service-name>\<aa>\<bb>\<cc>\
- # <aabbcc-rest-of-instance-guid>\<serialization data>
- def get_serialized_data_path(name)
- File.join(@config["serialization_base_dir"], "serialize", @config["service_name"] , name[0,2],name[2,2], name[4,2], name)
+ def generate_download_url(name, token)
+ service = @config["service_name"]
+ url_template = @config["download_url_template"]
+ eval "\"#{url_template}\""
+ end
end
- # Update the download token for a serialized file and save it in redis
- def update_download_token(service, name, file_name, token)
- key = "#{service}:#{name}:token"
- client.hset(redis_key(key), :token, token)
- client.hset(redis_key(key), :file, file_name)
- end
+ class BaseImportFromURLJob < SerializationJob
+ attr_reader :url, :temp_file_path
+ # Sub class should return true for a successful import job.
+ def perform
+ begin
+ required_options :service_id, :url
+ @name = options["service_id"]
+ @url = options["url"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ lock = create_lock
+ lock.lock do
+ @temp_file_path = File.join(@config["tmp_dir"], "#{name}")
+ FileUtils.rm_rf(temp_file_path)
+ fetch_url(url, temp_file_path)
+ raise ServiceError.new(ServiceError::BAD_SERIALIZED_DATAFILE, url) unless validate_input(temp_file_path)
+ result = execute
+ @logger.info("Results of import from url: #{result}")
+
+ job_result = { :result => :ok }
+ completed(Yajl::Encoder.encode(job_result))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ FileUtils.rm_rf(temp_file_path) if temp_file_path
+ end
+ end
- def parse_config
- @config = Yajl::Parser.parse(ENV['WORKER_CONFIG'])
- raise "Need environment variable: WORKER_CONFIG" unless @config
- end
+ # Fetch remote uri and stream content to file.
+ def fetch_url(url, file_path)
+ # TODO check the file size before download?
+ File.open(file_path, "wb+") do |f|
+ c = Curl::Easy.new(url)
+ c.on_body{|data| f.write(data)}
+ c.perform
+ end
+ end
- def cleanup(name)
- return unless name
- FileUtils.rm_rf(get_serialized_data_path(name))
end
- # Fetch remote uri and stream content to file.
- def fetch_url(url, file_path)
- # TODO check the file size before download?
- File.open(file_path, "wb+") do |f|
- c = Curl::Easy.new(url)
- c.on_body{|data| f.write(data)}
- c.perform
+ class BaseImportFromDataJob < SerializationJob
+ attr_reader :temp_file_path
+ # Sub class should return true for a successful import job.
+ def perform
+ begin
+ required_options :service_id, :temp_file_path
+ @name = options["service_id"]
+ @temp_file_path = options["temp_file_path"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ lock = create_lock
+ lock.lock do
+ raise "Can't find temp file: #{@temp_file_path}" unless File.exists? temp_file_path
+ raise ServiceError.new(SerivceError::BAD_SERIALIZED_DATAFILE, "request") unless validate_input(temp_file_path)
+
+ result = execute
+ @logger.info("Results of import from url: #{result}")
+
+ job_result = { :result => :ok }
+ completed(Yajl::Encoder.encode(job_result))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ FileUtils.rm_rf(@temp_file_path) if @temp_file_path
+ end
end
end
end
View
302 base/lib/base/job/snapshot.rb
@@ -1,144 +1,252 @@
# Copyright (c) 2009-2011 VMware, Inc.
-require "resque/job_with_status"
+require "resque-status"
require "fileutils"
require "vcap/logging"
+require "uuid"
$LOAD_PATH.unshift File.join(File.dirname(__FILE__), '..')
require "service_error"
-module VCAP
- module Services
- end
-end
+module VCAP::Services::Base::AsyncJob
+ module Snapshot
-module VCAP::Services::Snapshot
+ SNAPSHOT_KEY_PREFIX = "vcap:snapshot".freeze
+ SNAPSHOT_ID = "maxid".freeze
- SNAPSHOT_KEY_PREFIX = "vcap:snapshot".freeze
- SNAPSHOT_ID = "maxid".freeze
+ class << self
+ attr_reader :redis
- class << self
- attr_reader :redis, :logger
+ def redis_connect
+ @redis = ::Redis.new(Config.redis_config)
- # Config the redis using options
- def redis_connect(opts)
- resque = %w(host port password).inject({}){|res, o| res[o.to_sym] = opts[o]; res}
- @redis = ::Redis.new(resque)
+ redis_init
+ end
- redis_init
+ # initialize necessary keys
+ def redis_init
+ @redis.setnx("#{SNAPSHOT_KEY_PREFIX}:#{SNAPSHOT_ID}", 1)
+ end
end
- # Supply a redis instance
- def redis=(redis)
- raise "Snapshot requires redis configuration." unless redis
- @redis = redis
+ def client
+ Snapshot.redis
+ end
- redis_init
+ # Get all snapshots related to a service instance
+ #
+ def service_snapshots(service_id)
+ return unless service_id
+ res = client.hgetall(redis_key(service_id))
+ res.values.map{|v| Yajl::Parser.parse(v)}
end
- # initialize necessary keys
- def redis_init
- @redis.setnx("#{SNAPSHOT_KEY_PREFIX}:#{SNAPSHOT_ID}", 1)
+ # Return total snapshots count
+ #
+ def service_snapshots_count(service_id)
+ return unless service_id
+ client.hlen(redis_key(service_id))
end
- def logger=(logger)
- @logger = logger
+ # Get detail information for a single snapshot
+ #
+ def snapshot_details(service_id, snapshot_id)
+ return unless service_id && snapshot_id
+ res = client.hget(redis_key(service_id), snapshot_id)
+ Yajl::Parser.parse(res) if res
end
- end
- def client
- VCAP::Services::Snapshot.redis
- end
+ # Generate unique id for a snapshot
+ def get_snapshot_id
+ client.incr(redis_key(SNAPSHOT_ID)).to_s
+ end
- # Get all snapshots related to a service instance
- #
- def service_snapshots(service_id)
- return unless service_id
- res = client.hgetall(redis_key(service_id))
- res.values.map{|v| Yajl::Parser.parse(v)}
- end
+ def save_snapshot(service_id , snapshot)
+ return unless service_id && snapshot
+ msg = Yajl::Encoder.encode(snapshot)
+ client.hset(redis_key(service_id), snapshot[:snapshot_id], msg)
+ end
- # Get detail information for a single snapshot
- #
- def snapshot_details(service_id, snapshot_id)
- return unless service_id && snapshot_id
- res = client.hget(redis_key(service_id), snapshot_id)
- Yajl::Parser.parse(res) if res
- end
+ def delete_snapshot(service_id , snapshot_id)
+ return unless service_id && snapshot_id
+ client.hdel(redis_key(name), snapshot_id)
+ end
- # Generate unique id for a snapshot
- def get_snapshot_id
- client.incr(redis_key(SNAPSHOT_ID)).to_s
- end
+ def fmt_error(e)
+ "#{e}: [#{e.backtrace.join(" | ")}]"
+ end
- def save_snapshot(service_id , snapshot)
- return unless service_id && snapshot
- msg = Yajl::Encoder.encode(snapshot)
- client.hset(redis_key(service_id), snapshot[:snapshot_id], msg)
- end
+ protected
- def delete_snapshot(service_id , snapshot_id)
- return unless service_id && snapshot_id
- client.hdel(redis_key(name), snapshot_id)
- end
+ def redis_key(key)
+ "#{SNAPSHOT_KEY_PREFIX}:#{key}"
+ end
- protected
+ # common utils for snapshot job
+ class SnapshotJob
+ attr_reader :name, :snapshot_id
+ include Snapshot
+ include Resque::Plugins::Status
+ include VCAP::Services::Base::Error
+
+ class << self
+ def queue_lookup_key
+ :node_id
+ end
+
+ def select_queue(*args)
+ result = nil
+ args.each do |arg|
+ result = arg[queue_lookup_key]if (arg.is_a? Hash)&& (arg.has_key?(queue_lookup_key))
+ end
+ @logger = Config.logger
+ @logger.info("Select queue #{result} for job #{self.class} with args:#{args.inspect}") if @logger
+ result
+ end
+ end
- def redis_key(key)
- "#{SNAPSHOT_KEY_PREFIX}:#{key}"
- end
+ def initialize(*args)
+ super(*args)
+ parse_config
+ init_worker_logger
+ Snapshot.redis_connect
+ end
- # common utils for snapshot job
- class SnapshotJob < Resque::JobWithStatus
- include VCAP::Services::Snapshot
- include VCAP::Services::Base::Error
+ def init_worker_logger
+ @logger = Config.logger
+ end
- class << self
- attr_reader :logger
+ def required_options(*args)
+ missing_opts = args.select{|arg| !options.has_key? arg.to_s}
+ raise ArgumentError, "Missing #{missing_opts.join(', ')} in options: #{options.inspect}" unless missing_opts.empty?
+ end
- def queue_lookup_key
- :node_id
+ def create_lock
+ lock_name = "lock:lifecycle:#{name}"
+ lock = Lock.new(lock_name, :logger => @logger)
+ lock
end
- def logger=(logger)
- @logger = logger
+ # the snapshot path structure looks like <base-dir>\snapshots\<service-name>\<aa>\<bb>\<cc>\
+ # <aabbcc-rest-of-instance-guid>\snapshot_id\<service specific data>
+ def get_dump_path(name, snapshot_id)
+ File.join(@config["snapshots_base_dir"], "snapshots", @config["service_name"] , name[0,2],name[2,2], name[4,2], name, snapshot_id.to_s)
end
- def select_queue(*args)
- result = nil
- args.each do |arg|
- result = arg[queue_lookup_key]if (arg.is_a? Hash)&& (arg.has_key?(queue_lookup_key))
- end
- @logger.info("Select queue #{result} for job #{self.class} with args:#{args.inspect}") if @logger
- result
+ def parse_config
+ @config = Yajl::Parser.parse(ENV['WORKER_CONFIG'])
+ raise "Need environment variable: WORKER_CONFIG" unless @config
end
- end
- def initialize(*args)
- super(*args)
- parse_config
- init_worker_logger()
- end
+ def cleanup(name, snapshot_id)
+ return unless name && snapshot_id
+ @logger.info("Clean up snapshot and files for #{name}, snapshot id: #{snapshot_id}")
+ delete_snapshot(name, snapshot_id)
+ FileUtils.rm_rf(get_dump_path(name, snapshot_id))
+ end
- def init_worker_logger
- VCAP::Logging.setup_from_config(@config["logging"])
- @logger = VCAP::Logging.logger("#{@config["service_name"]}_worker")
+ def handle_error(e)
+ @logger.error("Error in #{self.class} uuid:#{@uuid}: #{fmt_error(e)}")
+ err = (e.instance_of?(ServiceError)? e : ServiceError.new(ServiceError::INTERNAL_ERROR)).to_hash
+ err_msg = Yajl::Encoder.encode(err)
+ failed(err_msg)
+ end
end
- # the snapshot path structure looks like <base-dir>\snapshots\<service-name>\<aa>\<bb>\<cc>\
- # <aabbcc-rest-of-instance-guid>\snapshot_id\<service specific data>
- def get_dump_path(name, snapshot_id)
- File.join(@config["snapshots_base_dir"], "snapshots", @config["service_name"] , name[0,2],name[2,2], name[4,2], name, snapshot_id.to_s)
+ class BaseCreateSnapshotJob < SnapshotJob
+ # workflow template
+ # Sub class should implement execute method which returns hash represents of snapshot like:
+ # {:snapshot_id => 1, :size => 100}
+ def perform
+ begin
+ required_options :service_id
+ @name = options["service_id"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ @snapshot_id = get_snapshot_id
+ lock = create_lock
+
+ lock.lock do
+ quota = @config["snapshot_quota"]
+ if quota
+ current = service_snapshots_count(name)
+ @logger.debug("Current snapshots count for #{name}: #{current}, max: #{quota}")
+ raise ServiceError.new(ServiceError::OVER_QUOTA, name, current, quota) if current >= quota
+ end
+
+ snapshot = execute
+ @logger.info("Results of create snapshot: #{snapshot.inspect}")
+
+ save_snapshot(name, snapshot)
+
+ job_result = { :snapshot_id => snapshot[:snapshot_id] }
+ completed(Yajl::Encoder.encode(job_result))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ cleanup(name, snapshot_id)
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ end
+ end
end
- def parse_config
- @config = Yajl::Parser.parse(ENV['WORKER_CONFIG'])
- raise "Need environment variable: WORKER_CONFIG" unless @config
+ class BaseDeleteSnapshotJob < SnapshotJob
+ def perform
+ begin
+ required_options :service_id, :snapshot_id
+ @name = options["service_id"]
+ @snapshot_id = options["snapshot_id"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ lock = create_lock
+
+ lock.lock do
+ result = execute
+ @logger.info("Results of delete snapshot: #{result}")
+
+ delete_snapshot(name, snapshot_id)
+
+ completed(Yajl::Encoder.encode({:result => :ok}))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ end
+ end
+
+ def execute
+ cleanup(name, snapshot_id)
+ end
end
- def cleanup(name, snapshot_id)
- return unless name && snapshot_id
- delete_snapshot(name, snapshot_id)
- FileUtils.rm_rf(get_dump_path(name, snapshot_id))
+ class BaseRollbackSnapshotJob < SnapshotJob
+ # workflow template
+ # Subclass implement execute method which returns true for a successful rollback
+ def perform
+ begin
+ required_options :service_id, :snapshot_id
+ @name = options["service_id"]
+ @snapshot_id = options["snapshot_id"]
+ @logger.info("Launch job: #{self.class} for #{name}")
+
+ lock = create_lock
+
+ lock.lock do
+ result = execute
+ @logger.info("Results of rollback snapshot: #{result}")
+
+ completed(Yajl::Encoder.encode({:result => :ok}))
+ @logger.info("Complete job: #{self.class} for #{name}")
+ end
+ rescue => e
+ handle_error(e)
+ ensure
+ set_status({:complete_time => Time.now.to_s})
+ end
+ end
end
end
end
View
146 base/lib/base/node.rb
@@ -24,12 +24,12 @@ def initialize(options)
z_interval = options[:z_interval] || 30
EM.add_periodic_timer(z_interval) do
EM.defer { update_varz }
- end
+ end if @node_nats
# Defer 5 seconds to give service a change to wake up
EM.add_timer(5) do
EM.defer { update_varz }
- end
+ end if @node_nats
end
def flavor
@@ -40,7 +40,7 @@ def on_connect_node
@logger.debug("#{service_description}: Connected to node mbus")
%w[provision unprovision bind unbind restore disable_instance
- enable_instance import_instance cleanup_nfs purge_orphan
+ enable_instance import_instance update_instance cleanupnfs_instance purge_orphan
].each do |op|
eval %[@node_nats.subscribe("#{service_name}.#{op}.#{@node_id}") { |msg, reply| EM.defer{ on_#{op}(msg, reply) } }]
end
@@ -134,46 +134,111 @@ def on_restore(msg, reply)
publish(reply, encode_failure(response, e))
end
- # disable and dump instance
+ # Disable and dump instance
def on_disable_instance(msg, reply)
@logger.debug("#{service_description}: Disable instance #{msg} request from #{reply}")
- credentials = Yajl::Parser.parse(msg)
- prov_cred, binding_creds = credentials
- instance = prov_cred['name']
- file_path = get_migration_folder(instance)
+ response = SimpleResponse.new
+ request = Yajl::Parser.parse(msg)
+ prov_handle, binding_handles = request
+ prov_cred = prov_handle["credentials"]
+ binding_creds = get_all_bindings(binding_handles)
+ file_path = get_migration_folder(prov_handle["service_id"])
FileUtils.mkdir_p(file_path)
result = disable_instance(prov_cred, binding_creds)
- result = dump_instance(prov_cred, binding_creds, file_path) if result
- publish(reply, Yajl::Encoder.encode(result))
+ if result
+ # Do dump together with disable for simpler migration logic
+ result = dump_instance(prov_cred, binding_creds, file_path)
+ if result
+ publish(reply, encode_success(response))
+ else
+ publish(reply, encode_failure(response))
+ end
+ else
+ publish(reply, encode_failure(response))
+ end
rescue => e
@logger.warn("Exception at on_disable_instance #{e}")
+ publish(reply, encode_failure(response, e))
end
- # enable instance and send updated credentials back
+ # Enable instance, the opposite operation of disable
def on_enable_instance(msg, reply)
@logger.debug("#{service_description}: enable instance #{msg} request from #{reply}")
- credentials = Yajl::Parser.parse(msg)
- prov_cred, binding_creds_hash = credentials
+ response = SimpleResponse.new
+ request = Yajl::Parser.parse(msg)
+ prov_handle, binding_handles = request
+ prov_cred = prov_handle["credentials"]
+ binding_creds_hash = get_all_bindings_with_option(binding_handles)
result = enable_instance(prov_cred, binding_creds_hash)
- # Update node_id in provision credentials..
- prov_cred, binding_creds_hash = result
- prov_cred['node_id'] = @node_id
- result = [prov_cred, binding_creds_hash]
- publish(reply, Yajl::Encoder.encode(result))
+ if result
+ publish(reply, encode_success(response))
+ else
+ publish(reply, encode_failure(response))
+ end
rescue => e
@logger.warn("Exception at on_enable_instance #{e}")
+ publish(reply, encode_failure(response, e))
+ end
+
+ # Import the generated data
+ def on_import_instance(msg, reply)
+ @logger.debug("#{service_description}: import instance #{msg} request from #{reply}")
+ response = SimpleResponse.new
+ request = Yajl::Parser.parse(msg)
+ prov_handle, binding_handles = request
+ prov_cred = prov_handle["credentials"]
+ binding_creds_hash = get_all_bindings_with_option(binding_handles)
+ plan = prov_handle["configuration"]["plan"]
+ file_path = get_migration_folder(prov_handle["service_id"])
+ result = import_instance(prov_cred, binding_creds_hash, file_path, plan)
+ if result
+ publish(reply, encode_success(response))
+ else
+ publish(reply, encode_failure(response))
+ end
+ rescue => e
+ @logger.warn("Exception at on_import_instance #{e}")
+ publish(reply, encode_failure(response, e))
+ end
+
+ # Update credentials in destination node of migration
+ def on_update_instance(msg, reply)
+ @logger.debug("#{service_description}: update instance #{msg} request from #{reply}")
+ request = Yajl::Parser.parse(msg)
+ prov_handle, binding_handles = request
+ prov_cred = prov_handle["credentials"]
+ binding_creds_hash = get_all_bindings_with_option(binding_handles)
+ result = update_instance(prov_cred, binding_creds_hash)
+ # Need decrease the capacity in destination node when finish migration
+ @capacity_lock.synchronize{ @capacity -= capacity_unit }
+ prov_cred, binding_creds_hash = result
+ # Update node_id in provision credentials
+ prov_cred["node_id"] = @node_id
+ handles = []
+ prov_handle["credentials"] = prov_cred
+ handles << prov_handle
+ binding_handles.each do |handle|
+ handle["credentials"] = binding_creds_hash[handle["service_id"]]["credentials"]
+ handles << handle
+ end
+ publish(reply, Yajl::Encoder.encode(handles))
+ rescue => e
+ @logger.warn("Exception at on_update_instance #{e}")
+ response = SimpleResponse.new
+ publish(reply, encode_failure(response, e))
end
# Cleanup nfs folder which contains migration data
- def on_cleanup_nfs(msg, reply)
+ def on_cleanupnfs_instance(msg, reply)
@logger.debug("#{service_description}: cleanup nfs request #{msg} from #{reply}")
+ response = SimpleResponse.new
request = Yajl::Parser.parse(msg)
- prov_cred, binding_creds = request
- instance = prov_cred['name']
- FileUtils.rm_rf(get_migration_folder(instance))
- publish(reply, Yajl::Encoder.encode(true))
+ prov_handle, _ = request
+ FileUtils.rm_rf(get_migration_folder(prov_handle["service_id"]))
+ publish(reply, encode_success(response))
rescue => e
- @logger.warn("Exception at on_cleanup_nfs #{e}")
+ @logger.warn("Exception at on_cleanupnfs_instance #{e}")
+ publish(reply, encode_failure(response, e))
end
# Send all handles to gateway to check orphan
@@ -206,7 +271,7 @@ def purge_orphan(oi_list,ob_list)
oi_list.each do |ins|
begin
@logger.debug("Unprovision orphan instance #{ins}")
- unprovision(ins,[])
+ @capacity_lock.synchronize{ @capacity += capacity_unit } if unprovision(ins,[])
rescue => e
@logger.debug("Error on purge orphan instance #{ins}: #{e}")
end
@@ -244,16 +309,25 @@ def get_migration_folder(instance)
<