Permalink
Fetching contributors…
Cannot retrieve contributors at this time
286 lines (200 sloc) 11.4 KB

Contributing to the Kive Project

If you like this project and want to make it better, help out. You could report a bug, or pitch in with some development work.

Bug Reports and Enhancement Requests

Please create issue descriptions on GitHub. Be as specific as possible. Which version are you using? What did you do? What did you expect to happen? Are you planning to submit your own fix in a pull request?

Development

You will need to follow all the installation instructions in the INSTALL file, then open the source code in a Python IDE. You will also need to install some packages to run the tests.

pip install -r requirements-dev.txt

If you want to see what's currently being worked on, check out the waffle board.

Performance Testing

It can be useful to track where time is spent when running a pipeline or a set of tests. Python comes with a profiler module:

python -m cProfile -s cumtime manage.py test --settings=kive.test_settings >timing.txt

Another option is to install the gprof2dot package with pip. Then you can generate a call graph with timing information:

python -m cProfile -o timing.dat manage.py test --settings=kive.test_settings \
&& (echo strip ; echo "sort cumtime" ; echo "stats 500") | python -m pstats timing.dat >timing.txt \
&& gprof2dot -f pstats timing.dat -o timing.dot

Deploying a Release

See the project wiki for instructions on how to start a production server. Once you have set up your production server, this is how to deploy a new release:

  1. Make sure the code works in your development environment. Run all the Javascript tests by opening kive/SpecRunner.html, and run all the Django unit tests.

    ./manage.py test --settings kive.settings_test_pg

  2. Check if the kiveapi package needs to update its version number by looking for new commits in the /api folder.

  3. Check that all the issues in the current milestone are closed.

  4. Check that all your code is committed to git, configure your settings file with a STATIC_ROOT of /path/to/git/Kive/static_root, and then build and package your static files.

     cd /path/to/git/Kive
     npm install
     cd kive
     ./manage.py collectstatic -c
     cd ..
     tar -czvf static_root.tar.gz static_root
    
  5. Create a release on Github. Use "vX.Y" as the tag, where X.Y matches the version on the milestone. If you have to redo a release, you can create additional releases with tags vX.Y.1, vX.Y.2, and so on. Mark the release as pre-release until you finish deploying it.

  6. Attach the static_root.tar.gz file to the release on GitHub. Attach it as a binary below the description, not as an attachment in the description.

  7. Check on the site that there are no active runs (as an administrator, go to the Runs page under the User portal, and click the lock to give yourself the ability to view all runs), then kill the fleet.

     ssh user@server
     ps aux|grep runfleet
     sudo kill -int <pid for runfleet>
    
  8. (Optional, but skip at your own peril!) Make a complete backup of the Kive installation. We use a tool called barman to backup PostgreSQL.

     sudo su barman
     barman backup main
     ./manage.py dumpdata --indent=4 > db_backup.json
    

    This covers everything stored in the database, including models and records automatically generated by Django itself. These files should be kept unchanged, as the records inside may be highly interdependent and any changes may cause system-wide problems.

    That doesn't cover everything in the system, however, as files tracked by Kive are stored on the filesystem, in the directory specified by MEDIA_ROOT in kive/settings.py. To preserve these files, make an exact copy of the following subdirectories:

    • CodeResources
    • Datasets
    • Logs
    • Sandboxes

    Do not restructure or rename anything in these folders: the file paths are stored in the database, so it's important to not let the files' actual locations become desynchronized from the stored locations.

    We wrote some rsync scripts to do the backup (not provided in the repo):

     sudo su kivefleet
     cd ~/bin
     ./backup_coderesources && ./backup_datasets && ./backup_logs && ./backup_sandboxes
    

    If you ever need to restore this backup, see "Restoring the system after something's gone wrong".

  9. Get the code from Github onto the server.

     ssh user@server
     cd /usr/local/share/Kive/kive
     sudo chgrp -R kive .. # Do this if other users also deploy.
     git fetch
     git checkout tags/vX.Y
    
  10. Check if you need to set any new settings by running diff kive/settings_default.py kive/settings.py.

  11. Migrate the database as described in the Creating Database Tables section of INSTALL.md, and deploy the static files:

     ssh user@server
     cd /usr/local/share/Kive/kive
     ./manage.py migrate
     grep STATIC_ROOT kive/settings.py
     cd /path/to/static/..
     sudo rm -Rf static
     sudo wget https://github.com/cfe-lab/Kive/releases/download/vX.Y/static_root.tar.gz -O static_root.tar.gz
     sudo tar -xzvf static_root.tar.gz
     sudo mv static_root static
     sudo rm static_root.tar.gz
    
  12. Launch the fleet. This assumes that kiveuser is the user account used to run the fleet. The su -l will run that user's login scripts that should activate any needed virtual environment:

     sudo su -l kiveuser
     cd /usr/local/share/Kive/kive
     ./manage.py runfleet </dev/null &
    
  13. Restart apache:

    sudo /usr/sbin/apachectl restart
    

    On CentOS 7, use systemctl:

    sudo systemctl restart httpd
    
  14. Update the Kive API library if needed.

    cd /usr/local/share/Kive/api
    cat setup.py  # look at the new version number
    pip show kiveapi  # compare with the version installed in Python 2
    sudo python setup.py install  # if needed
    pip3 show kiveapi  # compare with the version installed in Python 3
    sudo python3 setup.py install  # if needed
    
  15. Remove the pre-release flag from the release.

  16. Close the milestone for this release, create one for the next release, and decide which issues you will include in that milestone.

Restoring the system after something's gone wrong

If something goes wrong with the system, you can restore it using the backups created as per step 5 of the "Deploying a Release" section.

To do this, you must restore the system to the version it was in when the backup was made.
If you can use reverse migrations, then that's preferable, but depending on the specific migrations, this may fail. If so, then you can drop and re-create the database as per the instructions in INSTALL.md. From there, you can migrate forwards back to the state the system was in at the time of the backup.

Then, you can restore the data. First, flush the database:

./manage.py flush

(respond "yes" when it asks you whether to proceed or not). This will remove everything from the database, including records automatically generated by Kive or Django; we want to get rid of these as they may clash with the data in the backup. If your database backup file was called db_backup.json, you can then call

./manage.py loaddata db_backup.json

Lastly, clear out the four data subdirectories (CodeResources, Datasets, Logs, and Sandboxes), and replace them with your backed-up versions.

Unit tests

To run all the unit tests, run ./manage.py test. Note that running the full test suite can take around half an hour.

The front-end tests are run separately. run node tests-server.node.js in the root directory, which should open your browser. In this directory view, run both SpecRunner.html and SpecRunner_SystemJS.html.

Faster unit tests

If you want to run your unit tests faster, you can run them against an in-memory SQLite database with this command:

./manage.py test --settings kive.settings_test

This also reduces the amount of console output produced by the testing.

That still takes several minutes to run, so you may want to run a subset of the fastest tests: the mock tests. These tests don't access a database, so they are extremely fast. You can run them all with this command:

./manage.py test --settings kive.settings_mocked

Testing with a SQLite database may have slightly different behaviour from the PostgreSQL database, so you should occasionally run the tests with the default settings. Alternatively, to run the tests with all the default settings but with reduced console output:

./manage.py test --settings kive.settings_test_pg

All of these options disable some system tests that cover the parts of the Pipeline execution code that use Slurm. To enable the entire suite of tests:

./manage.py test --settings kive.settings_test_pg_slurm

These tests should be run before making a new release.

See the Django documentation for details on running specific tests.

If you want to time your unit tests to see which ones are slowest, install HotRunner.

sudo pip install unittest-xml-reporting

Then add these two lines to settings.py:

TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner'
TEST_OUTPUT_DIR = '/path/to/git/Kive/utils'

Finally, run the unit tests and the script to summarize them.

./manage.py test --settings kive.settings_test
./slow_test_report.py

Updating test fixtures

Fixtures are a Django feature which allow for test data to be persistently stored in the database during development, to avoid having to reload it every time a unit test is run. This is especially convenient for unit tests which involve actually running a pipeline, which can take a long time.

The fixtures which are used in our unit tests are created with the custom command ./manage.py update_test_fixtures. The code that is executed for this command can be found in portal/management/commands/update_test_fixtures.py, and the fixture files are in portal/fixtures. If this code or any functions it calls are modified, the fixtures will need to be re-created by running update_test_fixtures again.

Updating TypeScript and Sass files

Kive uses Webpack to bundle Javascript files. These bundles are generated on install, and should not be committed to the repository.

Be sure to adhere to Kive's TypeScript style by running grunt tslint on your contributions.

Run both the Webpack and Sass watchers simultaneously with npm run watch:all.

Updating embedded icon files

Some icon files are stored as base64-encoded strings which describe PNG images inside Javascript files.

Find the original icon in raw_assets in the project root. Make your modifications and then run grunt pngicons to compile them into the Javascript files. This command is also run on install.

* Recommended: If pngquant is available on your system, Grunt will use it to compress the icons.