Scottx611x/acceptance testing (#1534)

* Testing the feasibility of running multiple travis builds over different browsers/platforms * Wrap arg in quotes as to not evaluate spaces in env var * Try out an extra wait * I think testing Safari on anything other than OS X is a stretch * See if bumping OS X v. down helps * See if some extra waiting helps (I think Sauce lab's OS X machines are especially slow) * Move helper functions to utils file * Switch back o El Capitan because i believe timing was the only issue * move helper functions * Test adding some longer waits * Fix whitespace * fix imports * Comment out file upload test for now * Add `cleanup_on_error()` that we can decorate test methods with * Tidy up tests * Remove windows from acceptance testing * Move acceptance test running into own shell script and call from within travis * `cd` into the proper directory * `cd` back to `refinery` directory * Cleanup some more tests * Add id to aid testing * Add id to aid testing * Cleanup login method * Abstract out reused login moethod * Utilize relative import * PEP8 * only run acceptance tests if we detect a build for a PR * Always run our acceptance tests * Add acceptance tests for user registration and login * Don't use separate shell script (for now) * Remove `cd` comands * Refactor selenium utility functions * Add script to run acceptance tests on vagrant or travis * Lower `DEFAULT_WAIT` Now that we aren't relying on SauceLabs VMs * Cleanup setting of breakpoints, and refactor `cleanup_on_error` decorator * Remove unnecessary env var, and re-add `global` as to ensure a single build * Sauce_connect isn't needed * pip caching seems stable * PEP8 * Fail shell script if any step doesn't work properly * GeckoDriver expects a more recent version of Firefox * Remove Comments * Update Default wait time * Bump pytest-selenium version * Update Comment * Utilize `wait_until_id_clickable` * Add tests for global analysis popover * Add tests for Datasets panel * Update comment * Add a status to all generated Analyses * Update stats test to check for actual object counts * include gecko driver in our build process for ease of local selenium-based testing * Refactor old acceptance test implementation in favor of simplicity and uniformness with our current Django unit tests * Make virtual display bigger * Update selenium testing utils * Update assert body text method * Update selenium tests * Update `wait_until_id_visible()` * bump Factory-Boy requirement * Update travis as mentioned issue has been closed * Clean up gecko driver Puppet install * Update docstrings * Remove unnecessary tests * more geckodriver install cleanup * Change Default wait * update wait_until_id_visible() * pull out duplicated code into ui-deletion helper method * Update acceptance tests * Minor changes due to abstraction of deletion code * Fix typo * guest user creation isn't needed here anymore * Merge branch scottx611x/django_1.7_upgrade into scottx611x/acceptance_testing * Flip the order of some installed apps due to Django 1.7 ContentType loading * Decorate function as to not test 3rd party code's functionality * Utilize StaticLiveServerTestCase * Update delete_from_ui() helper * Refactor decorator name to be more verbose * Don't need to take a screenshot here * Add xvfb and python3 to travis * add pyvirtualdisplay req * Rename geckodriver class to selenium * Update order * Update order of INSTALLED_APPS for testing purposes * Update logging levels * Update TEST_RUNNER per Django system-check suggestion * Update tests * Fix `get_or_create()` * Access actual object returned from `get_or_create()` tuple * Update logger statement * Do not need to specify `live server` here since it has a default value * Update string formatting to work properly * Update isa/pre-isa archive deletion to produce less unnecessary errors * Update Tests * Manually save public group to sync db between threads (Specific to use of StaticLiveServerTestCase) * Update `DEFAULT_WAIT` * Share Datasets upon factory_boy creation * Create public group for APIV2 test case * Added group creation to the wrong TestCase * Remove `django-nose` in favor of Django's new: `DiscoverTestRunner` * This file shouldn't exist anymore. Probably wasn't caught in the merge here: 0c727cc * Include ontology fixture content inside of a Data migration (#1581) * Update puppet-archive version * Utilize `puppet-archive` and clean up neo4j.pp * Cleanup selenium manifest * Factor out solr class from init.pp * don't need to chmod here * We are now using StaticLiveServerTestCase * "Users of the `puppet-archive`module are responsible for archive package dependencies" * Remove unused var * Point to new archive download location * Fix string formatting error * Add `TEST_RUNNER` setting to base.py so that someone testing on `prod` locally could also benefeit * update order of `INSTALLED_APPS` addresses: #1353 * Refactor all tests that interact with the ORM to inherit from `TransactionTestCase` (removes need for crazy tearDown()'s) * DEFAULT_WAIT Shouldn't be that long * Manually save `public group` as to persist data between test and web driver threads * Update comment * Don't need to optimize sole's index in tests * Inherit from TransactionTestCase here * `20` was actually a good fail-safe if things get slow for one reason or another * switch order of INSTALLED APPS * Cleanup selenium test cases * Remove other `unittest` usages and unnecessary tests * Override setUp() in a cleaner manner * Add a travis fold * Clean up FileStoreitem and InvestigationLink error handling * Clean up error handling for isa & pre-isa archives * Update comments * Cleanup factory_boy factories and utils * This is already a default value * DRY * Update existing DataMigrations to a more easily manageable format * More robust Data migrations * Remove `pk` * RunPython is already encapsulated in a transaction * refactor `get_isa_archive()` & `get_pre_isa_archive()` * Analysis.status isn't necessary for testing * Add Error Handling to `SeleniumTestBase.setUp()` * Add `TEST_NON_SERIALIZED_APPS` setting due to pre Django 1.9 bug * Don't need to share datasets pubilcly in acceptance tests * Utilize `serialized_rollback` to ensure migration data persists in StaticLiveServerTestCases * Refactor isa/pre_isa archive fetching into one method * Remove other unnecessary instances of `tearDown()` * Update method signature * Update comment * Fix Test * Revert "Fix Test" This reverts commit 673827a. * Don't bundle (pre)isa_archive fetching functionality & Utilize existing `get_investigation()` * Handle potential AttributeErrors that could occcur * Old form was a better representation of our overall usecase * Reorder import
refinery-platform · Feb 28, 2017 · 42469a4 · 42469a4
1 parent 9563f68
commit 42469a4
Show file tree

Hide file tree

Showing 36 changed files with 1,239 additions and 1,915 deletions.
diff --git a/.travis.yml b/.travis.yml
@@ -1,24 +1,38 @@
 language: python
+cache: pip
+addons:
+  firefox: "49.0.1"
+  apt:
+    packages:
+      - xvfb
 python:
   - 2.7
-addons:
-  sauce_connect: true
-  # username and access_key provided as environment variables in travis config
 services:
   - postgresql
   - neo4j
 install:
   - pip install -r requirements.txt --quiet
+  # Install geckodriver required for selenium testing
+  - TAR=geckodriver-v0.11.1-linux64.tar
+  - wget https://github.com/mozilla/geckodriver/releases/download/v0.11.1/$TAR.gz
+  - gunzip $TAR.gz
+  - tar -xvf $TAR
+  - chmod a+x geckodriver
+  - export PATH=$PATH:`pwd`
 env:
-  - DJANGO_SETTINGS_MODULE=config.settings.dev REDUCE_TEST_OUTPUT=true PYTHONPATH=$PYTHONPATH:../refinery:../refinery/config
+  global:
+    # These env vars are available to every build
+    - REDUCE_TEST_OUTPUT=true
+    - PYTHONPATH=$PYTHONPATH:../refinery:../refinery/config
+    - DJANGO_SETTINGS_MODULE=config.settings.prod
+
 before_script:
   - createuser --createdb --no-superuser --no-createrole vagrant
   - createdb -O vagrant refinery
   - cd refinery
 # See http://www.stuartellis.eu/articles/erb/#running-erb-from-the-command-line
   - erb config/config.json.erb > config/config.json
   - python manage.py migrate --noinput
-  - python manage.py create_user 'guest' 'guest' 'guest@example.com' 'GuestFirst' 'GuestLast' 'GuestAffiliation' 'active'
   - npm install -g grunt-cli@0.1.13 bower@1.7.7 --progress false --quiet
   - cd ui
   - npm install --progress false --quiet
@@ -30,37 +44,20 @@ script:
   - flake8 --exclude=migrations,ui ..
 
   - echo 'travis_fold:start:grunt'
-  - pushd ui
-  - grunt
-  - popd
+  - pushd ui && grunt && popd
   - echo 'travis_fold:end:grunt'
 
-  - echo 'travis_fold:start:selenium'
-  # NOTE: Redirect runserver output since it's polluting logs
-  - python manage.py runserver > /dev/null 2>&1 &
-  - pushd ../selenium
-  # NOTE: Sauce Connect needs SAUCE_USERNAME and SAUCE_ACCESS_KEY
-  # (https://docs.travis-ci.com/user/sauce-connect/),
-  # while Pytest looks for SAUCELABS_USERNAME and SAUCELABS_API_KEY
-  # (http://pytest-selenium.readthedocs.io/en/latest/user_guide.html#sauce-labs).
-  # This has been recognized as a point of confusion, but there is no plan to change anything.
-  # https://github.com/pytest-dev/pytest-selenium/issues/53
-  - export SAUCELABS_USERNAME=$SAUCE_USERNAME SAUCELABS_API_KEY=$SAUCE_ACCESS_KEY
-  - printenv | sort | grep SAUCE | perl -pne 's/=(...).+/=$1.../' # Debug helper
-  - export BASE_URL=http://localhost:8000 CREDS_YML=guest_creds.yml UPLOAD=gff3.csv
-  - pytest --driver SauceLabs --capability browserName Firefox --capability tunnelIdentifier $TRAVIS_JOB_NUMBER -r fE
-  - popd
-  - echo 'travis_fold:end:selenium'
-
-  # If Django tests run first, the selenium test is truncated from the Travis log.
   - echo 'travis_fold:start:django-tests'
   - coverage run manage.py test
   - echo 'travis_fold:end:django-tests'
 
   - set +e # Currently, codecov does not always exit with 0, but that should not cause travis to fail.
+
 after_success:
+  - echo 'travis_fold:start:codecov'
   - codecov
   - npm run codecov
+  - echo 'travis_fold:end:codecov'
 notifications:
   slack:
     secure: nDs9Oj08nRizuD0edl6WcrSgaTPMyITQjZc4qPZpt+yOxUehWbrAmVhqYypfyvdj4qSi1E72rPTXftuBB1E1IZBgX4CCkrCkWGLgIxHaaValTd64oOX66eC3BbSehQxuJB7w1DWw54xBUkTy6+ufjAqiwhLpoEUeE296urAWYHU=
diff --git a/deployment/Puppetfile b/deployment/Puppetfile
@@ -8,6 +8,7 @@ mod 'puppetlabs/lvm', '0.7.0'
 mod 'puppetlabs/postgresql', '4.8.0'
 mod 'puppetlabs/rabbitmq', '5.6.0'
 mod 'puppetlabs/stdlib', '4.13.1'
+mod 'puppet-archive', '1.2.0'
 mod 'saz/timezone', '3.2.0'
 mod 'stankevich/python', '1.9.1'
 mod 'thias/sysctl', '1.0.2'

diff --git a/deployment/manifests/aws.pp b/deployment/manifests/aws.pp
@@ -28,7 +28,9 @@
 
 # See code in refinery-modules/refinery/...
 include refinery
-include refinery::pg
-include refinery::neo4j
 include refinery::apache2
 include refinery::aws
+include refinery::neo4j
+include refinery::pg
+include refinery::solr
+
diff --git a/deployment/manifests/default.pp b/deployment/manifests/default.pp
@@ -33,6 +33,9 @@
 
 # See code in refinery-modules/refinery/...
 include refinery
-include refinery::pg
-include refinery::neo4j
 include refinery::apache2
+include refinery::neo4j
+include refinery::pg
+include refinery::selenium
+include refinery::solr
+
diff --git a/deployment/refinery-modules/refinery/manifests/init.pp b/deployment/refinery-modules/refinery/manifests/init.pp
@@ -131,87 +131,6 @@
   ensure => directory,
 }
 
-class solr {
-  $solr_version = "5.3.1"
-  $solr_archive = "solr-${solr_version}.tgz"
-  $solr_url = "http://archive.apache.org/dist/lucene/solr/${solr_version}/${solr_archive}"
-
-  package { 'java':
-    name => 'openjdk-7-jdk',
-  }
-  exec { "solr_download":
-    command => "wget ${solr_url} -O ${solr_archive}",
-    cwd     => "/usr/local/src",
-    creates => "/usr/local/src/${solr_archive}",
-    path    => "/usr/bin",
-    timeout => 600,  # downloading can take a long time
-  }
-  ->
-  exec { "solr_extract_installer":
-    command => "tar -xzf ${solr_archive} solr-${solr_version}/bin/install_solr_service.sh --strip-components=2",
-    cwd     => "/usr/local/src",
-    creates => "/usr/local/src/install_solr_service.sh",
-    path    => "/bin",
-  }
-  ->
-  file { "${django_root}/solr/core/conf/solrconfig.xml":
-    ensure  => file,
-    content => template("${django_root}/solr/core/conf/solrconfig.xml.erb"),
-  }
-  ->
-  file { "${django_root}/solr/data_set_manager/conf/solrconfig.xml":
-    ensure  => file,
-    content => template("${django_root}/solr/data_set_manager/conf/solrconfig.xml.erb"),
-  }
-  ->
-  exec { "solr_install":  # also starts the service
-    command => "sudo bash ./install_solr_service.sh ${solr_archive} -u ${app_user}",
-    cwd     => "/usr/local/src",
-    creates => "/opt/solr-${solr_version}",
-    path    => "/usr/bin:/bin",
-    require => [ File["/opt"], Package['java'] ],
-  }
-  ->
-  file_line { "solr_config_home":
-    path  => "/var/solr/solr.in.sh",
-    line  => "SOLR_HOME=${django_root}/solr",
-    match => "^SOLR_HOME",
-  }
-  ->
-  file_line { "solr_config_log":
-    path  => "/var/solr/log4j.properties",
-    line  => "solr.log=${django_root}/log",
-    match => "^solr.log",
-  }
-  ~>
-  service { 'solr':
-    ensure     => running,
-    hasrestart => true,
-  }
-}
-include solr
-
-class solrSynonymAnalyzer {
-  $version = "2.0.0"
-  $url = "https://github.com/refinery-platform/solr-synonyms-analyzer/releases/download/v${version}/hon-lucene-synonyms.jar"
-
-  # Need to remove the old file manually as wget throws a weird
-  # `HTTP request sent, awaiting response... 403 Forbidden` error when the file
-  # already exists.
-
-  exec { "solr-synonym-analyzer-download":
-    command => "rm -f ${solr_lib_dir}/hon-lucene-synonyms.jar && wget -P ${solr_lib_dir} ${url}",
-    creates => "${solr_lib_dir}/hon-lucene-synonyms.jar",
-    path    => "/usr/bin:/bin",
-    timeout => 120,  # downloading can take some time
-    notify => Service['solr'],
-    require => Exec['solr_install'],
-  }
-}
-include solrSynonymAnalyzer
-
-
-
 include '::rabbitmq'
 
 class ui {

diff --git a/deployment/refinery-modules/refinery/manifests/neo4j.pp b/deployment/refinery-modules/refinery/manifests/neo4j.pp
@@ -94,12 +94,16 @@
   include owl2neo4j
 
   class neo4jPrePopulatedDB {
-    package { 'unzip':
-      name        => 'unzip',
-      ensure      => latest,
-      require     => Class['apt::update'],
+    $neo4j_user = "neo4j"
+    $neo4j_group = "nogroup"
+    $dirname = "graph.db"
+    $filename = "${dirname}.zip"
+    $install_path = "/var/lib/neo4j/data/"
+
+    package{"unzip":
+      ensure   => 'installed',
     }
-    ->
+
     exec { "stop neo4j service":
       command     => "sudo service neo4j-service stop",
       user        => $app_user,
@@ -116,22 +120,24 @@
       path        => ['/usr/bin/'],
     }
     ->
-    exec { "fetch pre-generated db":
-      command     => "sudo wget -q http://data.cloud.refinery-platform.org.s3.amazonaws.com/data/stem-cell-commons/neo4j/2015/graph.db.zip && yes | sudo unzip graph.db.zip",
-      cwd         => "/var/lib/neo4j/data/",
-      user        => $app_user,
-      group       => $app_group,
-      path        => ['/usr/bin/'],
-      timeout     => 1800,
+    archive { "fetch pre-generated db":
+      path        => "/tmp/${filename}",
+      source      => "http://data.cloud.refinery-platform.org.s3.amazonaws.com/data/stem-cell-commons/neo4j/2015/${filename}",
+      extract       => true,
+      extract_path  => "${install_path}",
+      creates       => "${install_path}/${dirname}",
+      cleanup       => true,
+      user        => 'root',
+      group       => 'root',
       require     => [
         Package['unzip'],
         Class['neo4jFetch'],
       ],
     }
     ->
-    exec { "change db ownership":
-      command     => "sudo chown -R neo4j:nogroup /var/lib/neo4j/data/graph.db",
-      path        => ['/usr/bin/', '/bin/'],
+    exec { 'neo4j permissions':
+      command   => "sudo chown -R $neo4j_user:$neo4j_group $install_path/$dirname",
+      path        => ['/usr/bin/'],
     }
     ->
     exec { "start neo4j service":
@@ -142,33 +148,6 @@
       require     => Service["neo4j-service"],
     }
     ->
-    exec { "fetch neo4j fixture":
-      command     => "sudo wget -q https://raw.githubusercontent.com/refinery-platform/ontology-imports/master/django-fixure-stemcellcommons.json",
-      cwd         => $django_root,
-      creates     => "$django_root/django-fixure-stemcellcommons.json",
-      path        => ['/usr/bin/'],
-      user        => $app_user,
-      group       => $app_group,
-      timeout     => 1800,
-    }
-    ->
-    exec { "install neo4j fixture":
-      command     => "${virtualenv}/bin/python manage.py loaddata django-fixure-stemcellcommons.json",
-      environment => ["DJANGO_SETTINGS_MODULE=${django_settings_module}"],
-      cwd         => $django_root,
-      user        => $app_user,
-      group       => $app_group,
-      # Exit code 1 will occur the second time this command is run. This
-      # will turn into a DataMigration soon but I'm avoiding creating more
-      # Migrations in Django 1.6 and will address this once Django 1.7 is
-      # Merged
-      returns     => [0,1],
-      require     => [
-        Python::Requirements[$requirements],
-        Postgresql::Server::Db["refinery"]
-      ],
-    }
-    ->
     exec { "install neo4j annotations":
       command     => "${virtualenv}/bin/python manage.py import_annotations -c",
       environment => ["DJANGO_SETTINGS_MODULE=${django_settings_module}"],
@@ -177,14 +156,6 @@
       group       => $app_group,
       require     => Exec['migrate'],
     }
-    ->
-    exec { "remove neo4j fixture":
-      command     => "rm -rf django-fixure-stemcellcommons.json",
-      cwd         => $django_root,
-      user        => $app_user,
-      group       => $app_group,
-      path        => ['/bin/'],
-    }
   }
   include neo4jPrePopulatedDB
 }
diff --git a/deployment/refinery-modules/refinery/manifests/selenium.pp b/deployment/refinery-modules/refinery/manifests/selenium.pp
@@ -0,0 +1,24 @@
+class refinery::selenium {
+  $geckodriver_version = 'v0.11.1'
+  $filename = "geckodriver-${geckodriver_version}-linux32.tar.gz"
+  $install_path = "/opt/geckodriver"
+
+  package { "firefox":}
+  package { "xvfb":}
+
+  archive { "fetch geckodriver":
+    path          => "/tmp/${filename}",
+    source     => "https://github.com/mozilla/geckodriver/releases/download/$geckodriver_version/$filename",
+    extract       => true,
+    extract_path  => '/opt/',
+    creates       => $install_path,
+    user        => 'root',
+    group       => 'root',
+    cleanup       => 'true',
+  }
+  ->
+  file { '/usr/bin/geckodriver':
+    ensure => 'link',
+    target => $install_path
+  }
+}