Merge branch 'next' into master

Conflicts: Rakefile Change-Id: I3795d9facffd3bfb3eff8acb32f8881511d793b5 Signed-off-by: Pin Xie <pxie@vmware.com>
cloudfoundry-attic · Jul 23, 2012 · fd8e00b · fd8e00b
2 parents 672845e + 8b1f9a2
commit fd8e00b
Show file tree

Hide file tree

Showing 22 changed files with 422 additions and 106 deletions.
diff --git a/Gemfile b/Gemfile
@@ -1,8 +1,9 @@
 source "http://rubygems.org"
+#source "http://ruby.taobao.org"
 
 gem "rake"
 gem "rspec"
-gem "cfoundry", "~> 0.2.0"
+gem "cfoundry", "~> 0.3.8"
 gem "vcap_logging", ">= 1.0"
 gem "interact"
 gem "curb"
@@ -19,3 +20,4 @@ gem "caldecott"
 group :ci do
   gem "ci_reporter"
 end
+gem "ruby-debug19"
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -2,6 +2,7 @@ GEM
   remote: http://rubygems.org/
   specs:
     addressable (2.2.8)
+    archive-tar-minitar (0.5.2)
     async_sinatra (0.5.0)
       rack (>= 1.2.1)
       sinatra (>= 1.0)
@@ -15,13 +16,14 @@ GEM
       em-http-request (~> 0.3.0)
       em-websocket (~> 0.3.1)
       uuidtools (~> 2.1.2)
-    cfoundry (0.2.2)
+    cfoundry (0.3.8)
       json_pure (~> 1.6.5)
       rest-client (~> 1.6.1)
       rubyzip (~> 0.9.5)
     ci_reporter (1.6.4)
       builder (>= 2.1.2)
     cli (1.1.1)
+    columnize (0.3.6)
     curb (0.8.0)
     diff-lcs (1.1.3)
     em-http-request (0.3.0)
@@ -39,6 +41,8 @@ GEM
       ruby-progressbar (~> 0.0.10)
     interact (0.4.2)
     json_pure (1.6.7)
+    linecache19 (0.5.12)
+      ruby_core_source (>= 0.1.4)
     mime-types (1.18)
     mongo (1.6.2)
       bson (~> 1.6.2)
@@ -60,7 +64,17 @@ GEM
       diff-lcs (~> 1.1.3)
     rspec-instafail (0.2.4)
     rspec-mocks (2.10.1)
+    ruby-debug-base19 (0.11.25)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby_core_source (>= 0.1.4)
+    ruby-debug19 (0.11.6)
+      columnize (>= 0.3.1)
+      linecache19 (>= 0.5.11)
+      ruby-debug-base19 (>= 0.11.19)
     ruby-progressbar (0.0.10)
+    ruby_core_source (0.1.5)
+      archive-tar-minitar (>= 0.5.2)
     rubyzip (0.9.8)
     sinatra (1.3.2)
       rack (~> 1.3, >= 1.3.6)
@@ -88,7 +102,7 @@ PLATFORMS
 DEPENDENCIES
   bson_ext
   caldecott
-  cfoundry (~> 0.2.0)
+  cfoundry (~> 0.3.8)
   ci_reporter
   cli
   curb
@@ -99,6 +113,7 @@ DEPENDENCIES
   progressbar (~> 0.11.0)
   rake
   rspec
+  ruby-debug19
   vcap_logging (>= 1.0)
   vmc
   yajl-ruby
diff --git a/README.md b/README.md
@@ -15,7 +15,7 @@ This repository contains tests for [vcap](https://github.com/cloudfoundry/vcap).
 1. Ruby 1.9.2
 2. Bundle >= 1.1
 3. admin user/password is needed for parallel run, if you don't have it, run serial like:
-```bundle exec rake tests[1]```
+```bundle exec rake full[1]```
 <br>It is recommended to manage Ruby runtime via RVM or rbenv
 
 ## _Tested Operating Systems_
@@ -24,7 +24,7 @@ This repository contains tests for [vcap](https://github.com/cloudfoundry/vcap).
 
 How to run it
 -------------
-1. ```git clone ssh://<YOUR-NAME>@reviews.cloudfoundry.org:29418/vcap-yeti```
+1. ```git clone git://github.com/cloudfoundry/vcap-yeti.git```
 2. ```cd vcap-yeti```
 3. ```./update.sh      ## this is not required for running administrative test cases```
 4. ```bundle exec rake full```
@@ -34,7 +34,6 @@ How to run it
     - admin user/admin password
    <br>This information is saved to ~/.bvt/config.yml file.
    <br>When run the second time around, Yeti will not prompt for the information again.
-6. If you want to see pending cases in run result: ```export VCAP_BVT_SHOW_PENDING=true```
 
 Notes:
 -----
@@ -55,14 +54,15 @@ Notes:
 |ACM_PASSWORD                 |acm user password    |ACM_PASSWORD=<***>                       |
 |SERVICE_BROKER_TOKEN         |service broker token |SERVICE_BROKER_TOKEN=<token>             |
 |SERVICE_BROKER_URL           |service broker url   |SERVICE_BROKER_URL=http://...            |
+|VCAP_BVT_LONGEVITY           |run testing N times  |VCAP_BVT_LONGEVITY=100                   |
 ```
 
 2. In order to support parallel running, and administrative test cases, Yeti will ask administrative
    account information.
    <br>However, yeti will not abuse administrative privileges, just list users, create users,
    <br>delete users created by the test script.
 3. rake tests/full use parallel by default, you could run in serial by specifying thread number=1:
-   ```bundle exec rake tests[1]```
+   ```bundle exec rake full[1]```
 4. As dev setup has limited resources, we strongly recommend running 1-4 threads against dev_setup.
 
 FAQ:
@@ -86,40 +86,55 @@ FAQ:
    steps, if any of the steps within an example fail then the test case will be marked as failed.
 
 4. Where are binary assets stored?
-   <br>A: Binary assets are stored in blobs.cloudfoundry.com which is a simple Sinatra application
-      with blob service backend.  These assets are then synchronized via the update.sh script into
-      the .assets-binaries directory of vcap-yeti.
+   <br>A: Binary assets are stored in http://blobs.cloudfoundry.com which is a simple Sinatra application
+      with blob service backend hosted on Cloud Foundry. These assets are then synchronized via the
+      update.sh script into the .assets-binaries directory of vcap-yeti.
 
 5. How do I submit binary assets?
-   <br>A: Currently binaries are generated manually based on source code updates to vcap-assets.  In
+   <br>A: Currently binaries are generated manually based on source code updates to vcap-assets. In
    the near future, source code updates will trigger a job to compile sources and update
    blobs.cloudfoundry.com.
 
 6. Where is the log file stored?
    <br>A: There two log files, runtime and error logs
       - Runtime log is stored in ~/.bvt/bvt.log
-      - Error log is stored in ~/.bvt/error.log
 
 7. What runtimes/frameworks/services should my environment have?
    <br>Dev setup:
-   - runtimes: java, ruby18, ruby19, node, node06, php, python2, erlangR14B01
+   - runtimes: java, ruby18, ruby19, node, node06, node08, php, python2, erlangR14B01
    - frameworks: java_web, sinatra, grails, rack, play, lift, spring, rails3, node, standalone, php,
    django, wsgi, otp_rebar
    - services: mongodb, mysql, postgresql, rabbitmq, redis, vblob, filesystem
    <br>(env variable CLOUD_FOUNDRY_EXCLUDED_COMPONENT can disable components, for details, please
-   check REAME under vcap/dev_setup.)
+   check README under vcap/dev_setup.)
 
    Dev instance:
-   - runtimes: java, java7, ruby18, ruby19, node, node06
+   - runtimes: java, java7, ruby18, ruby19, node, node06, node08
    - frameworks: java_web, sinatra, grails, rack, play, lift, spring, rails3, node, standalone
    - services: mongodb, mysql, postgresql, rabbitmq, redis, vblob
 
    Production:
-   - runtimes: java, java7, ruby18, ruby19, node, node06
+   - runtimes: java, java7, ruby18, ruby19, node, node06, node08
    - frameworks: java_web, sinatra, grails, rack, play, lift, spring, rails3, node, standalone
    - services: mongodb, mysql, postgresql, rabbitmq, redis
 
-   (updated on July 5th, 2012)
+   (updated on July 19th, 2012)
+
+8. Runtime errors
+   <br>Sometimes runtime errors happen during Yeti execution,
+   - 504 Gateway Error: Application fail to be started in 30 seconds
+   - 502 Internal Error: Application fail to connect to service instance, including provision/un-provision
+     bind/unbind.
+   - 404 Not Found: Route fail to redirect request to specific application URL
+
+9. Run specific case
+   <br>User can run specific case via passing spec file with specify line number of an example
+    or group as parameter
+    For example:
+    bundle exec rspec ./spec/simple/rails_console/ruby18_rails3_spec.rb:95
+
+10. Build java-based assets
+    <br>Please refer to docs/how-to-build-assets.md
 
 Rake Tasks:
 -----------
@@ -131,7 +146,6 @@ Rake Tasks:
 <br>run full tests in parallel, e.g. rake full\[5\] (default to 10, max = 16)
 - random
 <br>run all bvts randomly, e.g. rake random\[1023\] to re-run seed 1023
-- longevity loop bvt tests, e.g. rake longevity\[10\] (default to 100)
 - java
 <br>run java tests (spring, java_web) in parallel
 <br>e.g. rake java\[5\] (default to 10, max = 16)

diff --git a/Rakefile b/Rakefile
@@ -1,16 +1,19 @@
 $:.unshift(File.join(File.dirname(__FILE__), "lib"))
 require "harness"
+#require "harness/color_helper"
+include BVT::Harness::ColorHelpers
 
 task :default => [:help]
 
 desc "List help commands"
 task :help do
   puts "Usage: rake [command]"
   puts "  admin\t\trun admin test cases"
-  puts "  tests\t\trun core tests in parallel, e.g. rake test[5] (default to 10, max = 16)"
+  puts "  tests\t\trun core tests in parallel, e.g. rake test[5] (default to 10, max = 16)\n"
+  puts "       \t\tOptions: VCAP_BVT_LONGEVITY=N can loop this task (similiar to other tasks)\n"
+  puts "       \t\te.g. rake tests VCAP_BVT_LONGEVITY=10, rake ruby[4] VCAP_BVT_LONGEVITY=10"
   puts "  full\t\trun full tests in parallel, e.g. rake full[5] (default to 10, max = 16)"
   puts "  random\trun all bvts randomly, e.g. rake random[1023] to re-run seed 1023"
-  puts "  longevity\tloop bvt tests, e.g. rake longevity[10] (default to 100)"
   puts "  java\t\trun java tests (spring, java_web) in parallel\n" +
           "\t\te.g. rake java[5] (default to 10, max = 16)"
   puts "  jvm\t\trun jvm tests (spring, java_web, grails, lift) in parallel\n" +
@@ -32,11 +35,11 @@ task :full, :thread_number do |t, args|
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh("rspec --format Fuubar --color spec/ --tag ~admin")
+    longevity('sh("rspec --format Fuubar --color spec/ --tag ~admin")')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads, {"tags" => "~admin"})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads}, {'tags' => '~admin'})")
   end
 end
 
@@ -47,11 +50,11 @@ task :tests, :thread_number do |t, args|
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh("rspec --format Fuubar --color spec/ --tag ~admin --tag p1")
+    longevity('sh("rspec --format Fuubar --color spec/ --tag ~admin --tag p1")')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads, {"tags" => "p1,~admin"})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads}, {'tags' => 'p1,~admin'})")
   end
 end
 
@@ -60,11 +63,11 @@ task :random, :seed do |t, args|
   BVT::Harness::RakeHelper.generate_config_file
   BVT::Harness::RakeHelper.check_environment
   if args[:seed] != nil
-    sh "bundle exec rspec spec/ --tag ~admin --tag p1" +
-       " --seed #{args[:seed]} --format d -c"
+    longevity("sh 'bundle exec rspec spec/ --tag ~admin --tag p1' +
+       ' --seed #{args[:seed]} --format d -c'")
   else
-    sh "bundle exec rspec spec/ --tag ~admin --tag p1" +
-       " --order rand --format d -c"
+    longevity('sh "bundle exec rspec spec/ --tag ~admin --tag p1" +
+       " --order rand --format d -c"')
   end
 end
 
@@ -76,18 +79,18 @@ task :admin do
 end
 
 desc "Run java tests (spring, java_web)"
-task :java, :thread_number do |t, args|
+task :java, :thread_number, :longevity, :fail_fast do |t, args|
   threads = 10
   threads = args[:thread_number].to_i if args[:thread_number]
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh "bundle exec rspec --format Fuubar --color -P spec/**/*_spring_spec.rb," +
-     "spec/**/*_java_web_spec.rb"
+    longevity('sh "bundle exec rspec --format Fuubar --color -P spec/**/*_spring_spec.rb," +
+     "spec/**/*_java_web_spec.rb"')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads, {"pattern" => /_(spring|java_web)_spec\.rb/})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads}, {'pattern' => /_(spring|java_web)_spec\.rb/})")
   end
 end
 
@@ -98,13 +101,13 @@ task :jvm, :thread_number do |t, args|
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh "bundle exec rspec --format Fuubar --color -P spec/**/*_spring_spec.rb,spec" +
-     "/**/*_java_web_spec.rb,spec/**/*_grails_spec.rb,spec/**/*_lift_spec.rb"
+    longevity('sh "bundle exec rspec --format Fuubar --color -P spec/**/*_spring_spec.rb,spec" +
+     "/**/*_java_web_spec.rb,spec/**/*_grails_spec.rb,spec/**/*_lift_spec.rb"')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads,
-      {"pattern" => /_(spring|java_web|grails|lift)_spec\.rb/})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads},
+      {'pattern' => /_(spring|java_web|grails|lift)_spec\.rb/})")
   end
 end
 
@@ -115,12 +118,12 @@ task :ruby, :thread_number do |t, args|
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh "bundle exec rspec --format Fuubar --color -P spec/**/ruby18_*_spec.rb," +
-     "spec/**/ruby19_*_spec.rb"
+    longevity('sh "bundle exec rspec --format Fuubar --color -P spec/**/ruby18_*_spec.rb," +
+     "spec/**/ruby19_*_spec.rb"')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads, {"pattern" => /ruby1[89]_.+_spec\.rb/})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads}, {'pattern' => /ruby1[89]_.+_spec\.rb/})")
   end
 end
 
@@ -131,13 +134,13 @@ task :services, :thread_number do |t, args|
   if threads == 1
     BVT::Harness::RakeHelper.generate_config_file
     BVT::Harness::RakeHelper.check_environment
-    sh "bundle exec rspec --format Fuubar --color spec/ --tag mongodb --tag rabbitmq " +
-     "--tag mysql --tag redis --tag postgresql --tag neo4j --tag vblob"
+    longevity('sh "bundle exec rspec --format Fuubar --color spec/ --tag mongodb --tag rabbitmq " +
+     "--tag mysql --tag redis --tag postgresql --tag neo4j --tag vblob"')
   else
     BVT::Harness::RakeHelper.generate_config_file(true)
     BVT::Harness::RakeHelper.check_environment
-    BVT::Harness::ParallelHelper.run_tests(threads, {"tags" =>
-      "~admin,mongodb,rabbitmq,mysql,redis,postgresql,neo4j,vblob"})
+    longevity("BVT::Harness::ParallelHelper.run_tests(#{threads}, {'tags' =>
+      '~admin,mongodb,rabbitmq,mysql,redis,postgresql,neo4j,vblob'})")
   end
 end
 
@@ -151,34 +154,30 @@ task :sync_assets do
   BVT::Harness::RakeHelper.sync_assets
 end
 
-desc "longevity bvt tests"
-task :longevity, :looptimes do |t, args|
-  loop_times = 100
-  time_start = Time.now
-  if args[:looptimes] != nil && args[:looptimes].to_i > 0
-    loop_times = args[:looptimes].to_i
+def get_longevity_time
+  return ENV['VCAP_BVT_LONGEVITY'].to_i if ENV['VCAP_BVT_LONGEVITY']
+  return 1
+end
+
+def longevity(cmd)
+  loop_times = get_longevity_time
+  if loop_times == 1
+    eval(cmd)
+    return
+  elsif loop_times < 1
+    puts red("longevity input error")
+    return
   end
-  puts "loop times: #{loop_times}"
+  time_start = Time.now
+  puts yellow("loop times: #{loop_times}")
   $stdout.flush
   loop_times.times {|i|
-    BVT::Harness::RakeHelper.generate_config_file
-    BVT::Harness::RakeHelper.check_environment
-    cmd = "bundle exec rspec --format Fuubar --color spec/ --tag ~admin --tag p1"
-    output = %x[#{cmd}]
-    puts output
-    if output.include? "Failures:"
-      t1 = Time.now
-      running_time = (t1 - time_start).to_i
-      puts "Task failed!"
-      puts "longevity task has succeeded for #{i} times"
-      puts "longevity task has been running for #{running_time} seconds"
-      break
-    end
-    t2 = Time.now
-    running_time = (t2 - time_start).to_i
-    puts "Task succeeded!"
-    puts "longevity task has succeeded for #{i+1} times"
-    puts "longevity task has been running for #{running_time} seconds"
-    $stdout.flush
+    puts yellow("This is #{i} run.")
+    eval(cmd)
   }
+  puts yellow("longevity finished!")
+  puts yellow("loop times:    #{loop_times}")
+  t1 = Time.now
+  running_time = (t1 - time_start).to_i
+  puts yellow("total running time: #{running_time} seconds")
 end