Overwatch 1.2.1

[1.2.1] - 12 November 2018

• 1.2.1 fixes a tagging issue. Below is the 1.2 changelog.

Added

• Travis CI to run tests, flake8, and coverage. Also added coveralls to expand of tests coverage information. Note that the flake8 tests run in the Travis CI python 2.7 and 3.6, while the tests run inside the docker images.
• Travis CI will build new docker images on each commit, which are known as rehlers/overwatch:latest-py*. Tagged releases will be available as rehlers/overwatch:tag-py*.
• Releases are automatically made on PyPI through Travis CI.
• Added overwatch.base.dataTransfer module, which is responsible for transferring data provided by the receiver to various Overwatch and EOS sites. This modules is fairly well covered by tests.
• Added webApp, processing, and data transfer monitoring via sentry. It hooks into everything (exceptions, logs, etc) to help identify and debug issues.
• Added a module for replaying data in overwatch.base.replay. Can be used to generically replay processed data, moving from one directory to another. For further information, see the README in overwatch.base.
• Units tests for timestamp extraction in overwatch.base.utilities. See: 62737f1.
• Added some integration tests for creating runs and subsystems in overwatch.processing.processRuns. See: b09e738.
• Added rudimentary ZMQ receiver monitoring via the dataTransfer module. If no files have been transferred in 12 hours, a warning will be emitted. See: 4ea80cf2.
• Added dedicated direct ZMQ receiver monitor via the overwatch.receiver.monitor module. It monitors heartbeat information directly from the ZMQ receivers. It is a supplement to the monitoring via the dataTransfer module. See: c081a8b.
• repr and str methods to most of the processing classes to aid in debugging. See: 3a544d6 and 43657bc.

Changed

• Updated overwatchDeploy to be class based, and generally far more stable and extensible. It is also broadly covered by unit tests.
• Improved webApp status information (and removed obsolete code). See: c41e559.
• Changed all time related functionality to utilize the pendulum package. It makes live so much easier! See: 8fe66ba.

Fixed

• Creation of run and subsystem containers as new data arrives. Issues were caused by received files arriving at different times, which split up the processing. See: b9230b9, with fixes in 1cfb18c, 5e3630a, f7863a6, and 364543f.
• Fixed data transfer to only select on files which end in ".root". ROOT appears to create temporary files when writing which are sometimes picked up during data transfer. See: 83412bb.
• Execution data is now stored in the exec directory. Information includes logs, configurations (except for the Overwatch config, which must be in the executing directory), and more sensitive files (SSL, etc). They will be used automatically in the docker images and by supervisor This was changed to better reflect what information was stored. See: ec64fbd.
• Removed the deploy directory, along with much of its obsolete contents, which have been replaced by the deploy module. See: 9735748.
• Timestamp handling was inconsistent, which caused problems when trying to improve the isRunOngoing(...) logic. Switch to explicitly handling the time zones with the pendulum package, which makes things much easier. See: 819fa6a for the start, and 8fe66ba for the last commit, which resolved all known issues.
• The docker images now run in a less privileged user.
• Ensure that the data replay copies to the proper folder when there is already an existing folder. See: f71afab.
• A wide variety of typos.