Contributing to the Kive Project
If you like this project and want to make it better, help out. You could report a bug, or pitch in with some development work.
Bug Reports and Enhancement Requests
Please create issue descriptions on GitHub. Be as specific as possible. Which version are you using? What did you do? What did you expect to happen? Are you planning to submit your own fix in a pull request?
You will need to follow all the installation instructions in the INSTALL file, then open the source code in a Python IDE. You will also need to install some packages to run the tests.
pip install -r requirements-dev.txt
If you want to see what's currently being worked on, check out the waffle board.
It can be useful to track where time is spent when running a pipeline or a set of tests. Python comes with a profiler module:
python -m cProfile -s cumtime manage.py test --settings=kive.test_settings >timing.txt
Another option is to install the gprof2dot package with pip. Then you can generate a call graph with timing information:
python -m cProfile -o timing.dat manage.py test --settings=kive.test_settings \ && (echo strip ; echo "sort cumtime" ; echo "stats 500") | python -m pstats timing.dat >timing.txt \ && gprof2dot -f pstats timing.dat -o timing.dot
Deploying a Release
See the project wiki for instructions on how to start a production server. Once you have set up your production server, this is how to deploy a new release:
kive/SpecRunner.html, and run all the Django unit tests.
./manage.py test --settings kive.settings_test_pg
Check if the kiveapi package needs to update its version number by looking for new commits in the
Check that all the issues in the current milestone are closed.
Check that all your code is committed to git, configure your settings file with a
/path/to/git/Kive/static_root, and then build and package your static files.
cd /path/to/git/Kive npm install cd kive ./manage.py collectstatic -c cd .. tar -czvf static_root.tar.gz static_root
Create a release on Github. Use "vX.Y" as the tag, where X.Y matches the version on the milestone. If you have to redo a release, you can create additional releases with tags vX.Y.1, vX.Y.2, and so on. Mark the release as pre-release until you finish deploying it.
static_root.tar.gzfile to the release on GitHub. Attach it as a binary below the description, not as an attachment in the description.
Check on the site that there are no active runs (as an administrator, go to the Runs page under the User portal, and click the lock to give yourself the ability to view all runs), then kill the fleet.
ssh user@server ps aux|grep runfleet sudo kill -int <pid for runfleet>
(Optional, but skip at your own peril!) Make a complete backup of the Kive installation. We use a tool called barman to backup PostgreSQL.
sudo su barman barman backup main ./manage.py dumpdata --indent=4 > db_backup.json
This covers everything stored in the database, including models and records automatically generated by Django itself. These files should be kept unchanged, as the records inside may be highly interdependent and any changes may cause system-wide problems.
That doesn't cover everything in the system, however, as files tracked by Kive are stored on the filesystem, in the directory specified by
kive/settings.py. To preserve these files, make an exact copy of the following subdirectories:
Do not restructure or rename anything in these folders: the file paths are stored in the database, so it's important to not let the files' actual locations become desynchronized from the stored locations.
We wrote some rsync scripts to do the backup (not provided in the repo):
sudo su kivefleet cd ~/bin ./backup_coderesources && ./backup_datasets && ./backup_logs && ./backup_sandboxes
If you ever need to restore this backup, see "Restoring the system after something's gone wrong".
Get the code from Github onto the server.
ssh user@server cd /usr/local/share/Kive/kive sudo chgrp -R kive .. # Do this if other users also deploy. git fetch git checkout tags/vX.Y
Check if you need to set any new settings by running
diff kive/settings_default.py kive/settings.py.
Migrate the database as described in the Creating Database Tables section of INSTALL.md, and deploy the static files:
ssh user@server cd /usr/local/share/Kive/kive ./manage.py migrate grep STATIC_ROOT kive/settings.py cd /path/to/static/.. sudo rm -Rf static sudo wget https://github.com/cfe-lab/Kive/releases/download/vX.Y/static_root.tar.gz -O static_root.tar.gz sudo tar -xzvf static_root.tar.gz sudo mv static_root static sudo rm static_root.tar.gz
Launch the fleet. This assumes that
kiveuseris the user account used to run the fleet. The
su -lwill run that user's login scripts that should activate any needed virtual environment:
sudo su -l kiveuser cd /usr/local/share/Kive/kive ./manage.py runfleet </dev/null &
sudo /usr/sbin/apachectl restart
On CentOS 7, use
sudo systemctl restart httpd
Update the Kive API library if needed.
cd /usr/local/share/Kive/api cat setup.py # look at the new version number pip show kiveapi # compare with the version installed in Python 2 sudo python setup.py install # if needed pip3 show kiveapi # compare with the version installed in Python 3 sudo python3 setup.py install # if needed
Remove the pre-release flag from the release.
Close the milestone for this release, create one for the next release, and decide which issues you will include in that milestone.
Restoring the system after something's gone wrong
If something goes wrong with the system, you can restore it using the backups created as per step 5 of the "Deploying a Release" section.
To do this, you must restore the system to the version it was in when the backup was made.
If you can use reverse migrations, then that's preferable, but depending on the specific migrations, this may fail. If so, then you can drop and re-create the database as per the instructions in INSTALL.md. From there, you can migrate forwards back to the state the system was in at the time of the backup.
Then, you can restore the data. First, flush the database:
(respond "yes" when it asks you whether to proceed or not). This will remove everything from the
database, including records automatically generated by Kive or Django; we want to get rid of these
as they may clash with the data in the backup. If your database backup file was called
db_backup.json, you can then call
./manage.py loaddata db_backup.json
Lastly, clear out the four data subdirectories (
and replace them with your backed-up versions.
To run all the unit tests, run
./manage.py test. Note that running the
full test suite can take around half an hour.
The front-end tests are run separately. run
node tests-server.node.js in the
root directory, which should open your browser. In this directory view, run both
SpecRunner.html and SpecRunner_SystemJS.html.
Faster unit tests
If you want to run your unit tests faster, you can run them against an in-memory SQLite database with this command:
./manage.py test --settings kive.settings_test
This also reduces the amount of console output produced by the testing.
That still takes several minutes to run, so you may want to run a subset of the fastest tests: the mock tests. These tests don't access a database, so they are extremely fast. You can run them all with this command:
./manage.py test --settings kive.settings_mocked
Testing with a SQLite database may have slightly different behaviour from the PostgreSQL database, so you should occasionally run the tests with the default settings. Alternatively, to run the tests with all the default settings but with reduced console output:
./manage.py test --settings kive.settings_test_pg
All of these options disable some system tests that cover the parts of the Pipeline execution code that use Slurm. To enable the entire suite of tests:
./manage.py test --settings kive.settings_test_pg_slurm
These tests should be run before making a new release.
See the Django documentation for details on running specific tests.
If you want to time your unit tests to see which ones are slowest, install HotRunner.
sudo pip install unittest-xml-reporting
Then add these two lines to
TEST_RUNNER = 'xmlrunner.extra.djangotestrunner.XMLTestRunner' TEST_OUTPUT_DIR = '/path/to/git/Kive/utils'
Finally, run the unit tests and the script to summarize them.
./manage.py test --settings kive.settings_test ./slow_test_report.py
Updating test fixtures
Fixtures are a Django feature which allow for test data to be persistently stored in the database during development, to avoid having to reload it every time a unit test is run. This is especially convenient for unit tests which involve actually running a pipeline, which can take a long time.
The fixtures which are used in our unit tests are created with the
./manage.py update_test_fixtures. The code that is
executed for this command can be found in
portal/management/commands/update_test_fixtures.py, and the fixture files
portal/fixtures. If this code or
any functions it calls are modified, the fixtures will need to be
re-created by running
Updating TypeScript and Sass files
Be sure to adhere to Kive's TypeScript style by running
grunt tslint on your
Run both the Webpack and Sass watchers simultaneously with
npm run watch:all.
Updating embedded icon files
Find the original icon in
raw_assets in the project root. Make your modifications
and then run
command is also run on install.
* Recommended: If
pngquant is available on your system, Grunt will use
it to compress the icons.