Log sample generation to merlin_info #234

ben-bay · 2020-06-11T23:49:52Z

Addresses #117

merlin/study/study.py

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* 1.7.0 (LLNL#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (LLNL#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (LLNL#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (LLNL#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (LLNL#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (LLNL#218) race conditions. * updated timeout (LLNL#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (LLNL#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (LLNL#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (LLNL#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (LLNL#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (LLNL#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (LLNL#229) * 1.6.1 * bugfix (LLNL#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (LLNL#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (LLNL#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (LLNL#236) will not alter the lsf launch. * Bugfix for new celery versions (LLNL#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (LLNL#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (LLNL#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (LLNL#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (LLNL#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (LLNL#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (LLNL#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (LLNL#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (LLNL#251) * Feature/ben/pgen (LLNL#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (LLNL#254) * Update CHANGELOG.md * pretty yaml (LLNL#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (LLNL#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (LLNL#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (LLNL#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (LLNL#256) * expose celery (LLNL#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (LLNL#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (LLNL#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (LLNL#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 * Refactor/ben/spec logic (#267) * improved lots of logic * added minimum format test * added to CHANGELOG * added to CHANGELOG * added 3 cli tests * added shortcut var * sorted imports * removed erroneous note from doc * null spec updates (for paper) (#270) * made samples generation actually null; now uses touch, once only * improving samples * updated job launcher * modifications * added echo * added echo * added times * fixed slurm command logic * script improvements to accomodate multiple node logs * read output script updates * read output script fixes * updated times * updated times * script updates * script changes * launcher script improvement: added run id to queue names * tweak * added account logic for quartz and pascal * path * fixed bug that mixed queues at w=32 and w=64 * bugfix * tweak * added new launcher specs * added 24 hour cap * fixed sample vals * script update * launcher changes * updated merlin monitor call * updates * fixed style * Feature/koning/maestro remove submit message (#269) * Add a MerlinStepRecord to remove Maestro message when tasks resubmit. * Add the dependent objects for MerlinStepRecord. * Add CHANGELOG entry for bugfix. * Revert quick fix changes. * fixed style * preparing run * script updates * updated launcher script: automated node list vals * removed predefined node lists * Update CHANGELOG.md * Update CHANGELOG.md * Feature/ben/null chain (#272) * added null chain scripts * spec improvements * removed sim spec, as null spec can now be customized to wait longer * added to merlin monitor * specified copied file name * added time to null_chain * max sample ramp * added to CHANGELOG * fixed style * 1.7.4 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com>

* Squash and merge develop to Main * Master (#2) * 1.7.0 (#257) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.1 hotfix (#261) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#264) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.2 (#265) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * 1.7.3 (#266) * locked maestro version * incremented to 1.5.3 * Update CHANGELOG.md * Feature/koning/vhost no leading slash (#217) * The broker name can now be amqps (with ssl) or amqp (without ssl). The default rabbitmq vhost is now <user> instead of /<user>. * Undo fix typos. * Fix spacing. * Add amqps conn for self doc. * Add inital running workers check to monitor. This should eliminate any (#218) race conditions. * updated timeout (#223) * updated default task timeout * Added to CHANGELOG * Feature/koning/create encrypt key init (#219) * Add init_key to create the encryption key when merlin config is run. * Move import to point of use for encryption. * Feature/koning/monitor upgrade (#221) * Fix sleep variable. * Change another sleep variable. * Change brackets. * Fix clarity for jobs but no consumers condition. * Fix comment. * Update comment. * Move check_merlin_status to router. * Run fix-style. * Bugfix/ben/expand name (#220) * working on bugfix * achieved desired behavior * made expansion anf filewriting conditional * fixed style * updated CHANGELOG * fixed samples and restart * fixed style * used variable * added name expansion test * temporarily commented out one test * fixed style * added debug block * changes * corrected bug * updated example; scripts are now copied into merlin_info * Feature/ben/expose visibility timeout (#224) * exposed visibility timeout in config file * added visibility timeouts to default config files * fixed style * added seconds to clarify name * fixed attribute name * locked celery version at 4.4.2 * updated CHANGELOG * v1.5.4 * adjusted CHANGELOG * adjusted CHANGELOG * Feature/koning/info conn timeout (#226) * Add a timeout check for the merlin info kombu connection test. The redis server will not comply with the connection_timeout config. * Update docs. * Load default timeout if non in config file (#229) * 1.6.1 * bugfix (#232) * bugfix * removed repetition in logic * Log sample generation to merlin_info (#234) * sample generation is now logged to merlin_info * Added to CHANGELOG * fixed style * Maestro 1.1.8+ update (#233) * working on adding maestro's schema validation to merlin * updated schemas * adding merlin_schema to spec logic * added debug prints * added print * fixing vestigial Maestro calls * adjusted merlin_schemas * fixed json file * validation progress * added from_dict replacement logic * adjusted maestrowf version req * fixed string * schema improvements * merlinsection schema updates * merlin section error parsing appears to work * updated CHANGELOG * fixed merge error * Added walltime to json schema * Add the bank and walltime keywords to the batch slurm launch, these (#236) will not alter the lsf launch. * Bugfix for new celery versions (#231) * Change the expand_tasks_with_samples task to use an immutable signature, no return code is neccessary. * Change only expand_tasks_with_samples to si() * Increase celery version requirement to the latest (4.4.5). * Update CHANGELOG. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * hidden test specs (#237) * added tests to run_tests.py * updated merlinsection schema * added full_spec test * added to CHANGELOG * merlinspec change * use regex to find variable tokens (#239) * now using regex to find variable tokens * added shell ref function * added to CHANGELOG * Revert maestro 1.1.8 support (#241) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * fixed cli tests, reactivated 3 old ones (#242) * fixed cli tests, reactivated 3 old ones * added to CHANGELOG * More provenance specs + logic improvements (#240) * added orig.yaml and expanded.yaml * added partial provenance spec * fixed style * added to CHANGELOG * removed comments * made env replacement more precise * fixed restart bug * changed provenance file names * logic update * fixed style * expansion now working * added cli test logic to test all 3 provenance file types * updated CHANGELOG * expansion sans io (#243) * reverted support of Maestro 1.1.8 * removed tests that use maestro validation * fixed style * added sections property to spec * reducing number of file writes and reads * provenance specs are correct * fixed style * removed unused imports * cleaning up logic * fixed style * final logic improvements * removed unused property * updated CHANGELOG * v1.6.2 * debugging study * added provenance tests * fixed path in test * added check for workflow name expanding to invalid filename * Docs/ben/level max dirs (#250) * added to documentation * added to CHANGELOG * added default * updated docs to reflect recent provenance spec update (#251) * Feature/ben/pgen (#248) * added basic pgen support * added to CHANGELOG * pgen, pargs, and env appear to work * added to CHANGELOG * fixed typo * added pgen cli test * added faq entry on pgen * Update Dockerfile (#254) * Update CHANGELOG.md * pretty yaml (#252) * starting * building pretty_dump function * updates * fixed style * tweaks * valid yaml * added yaml_sections * added to CHANGELOG * removed old yaml representer code * added cli test for equality of provenance specs * fixed pgen test * disabled test while not working * fixed yaml lists for source, paths, git * tweaks / fixes * env variable bugfix (#247) * removed expansion of env variables in provenance specs * added to CHANGELOG * corrected unit tests * added todos * working on expand_env_var function * fixed type bug * removed comments, prints, fixed style * added to CHANGELOG * tweaks * fixed output path * allowed study name to use env vars * fixed style * added to CHANGELOG, added support for 'restart' * Update CHANGELOG.md * added docs strings * added to docs * updated comment * fixed CHANGELOG * removed debug print * flux examples fix (#253) * Update flux example workflows to use new kvs query methods. * Update CHANGELOG with fix info. * Add coker fix comment. * Update CHANGELOG.md * added gitlab ci file * improved gitlab ci setup * renamed file * hotfix to allow distributed cli tests to run * redis update (#255) * Update redis to new version with TLS support. * Add changelog. * Bugfix for rediss cert_reqs keyword. (#256) * expose celery (#245) * added default app.yaml, still in progress... * added celery defaults * Update app.yaml * config override appears to be working * corrected value name * shifted some properties to be optional defaults * removed unneeded file * improvements * fixed style * fixed style * hid 'no override' message * tweaked var names * added 2 tests for celery app (#246) * fixed test * fixed workers bug * fixed unit tests for running offline * removed 2 unit tests * fixed CHANGELOG * added info print of all celery configurations being overridden * 1.7.0 * 1.7.0 * copy samplesfile to merlin_info (#259) * copy samplesfile to merlin_info * added to CHANGELOG * Bugfix/koning/process exceptions (#260) * Add bugfix log. * Add an exception catcher for the multiprocessing Process class. * fixed style * 1.7.1 * hotfix * fix * 1.7.2 * Update CHANGELOG.md * Update CHANGELOG.md * 1.7.3 * 1.7.3 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> * Revert "Master (#2)" This reverts commit 1d7495b. * [Flake8 src refactor] display.py and broker.py refactored. * [Standardize CI linting and Makefile] Updated the Makefile and CI to hold one standard for linting from setup.cfg. * Add ability to enter step walltime in HH:MM:SS format. With HH and MM optional. Previously, script adapters had assumed a format that depends on the batch type. Now it'll get converted internally to the right format. * Updated CHANGELOG and docs * Add testing for integers too * An explicit conversion test. Making mypy and black happy. * Move to 1.8.0 * Explictly wrap queue name string in quotes to make it more robust (#320) * Explictly wrap queue name string in quotes to make it more robust * Move CHANGELOG note to v1.8.0 * Update CHANGELOG.md * Update test_time_formats.py Utilized typing for all variables, utilizing PyTest parametrized marks to allow testing to flow regardless of individual fail states of test data. * Update test_time_formats.py Added test to verify some exception handling, compressed tests to not have as much indirection. * Typo in variable name change * Fix style before merging, update to Makefile so fix-style upgrades all dev packages before style check. (#321) Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> * Revert "Merge branch 'main' into develop" using -m option. This reverts commit a9276b8, reversing changes made to 63a094b. Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: fixdocker <68359629+fixdocker@users.noreply.github.com> Co-authored-by: Alexander Cameron Winter <winter27@quartz1148.llnl.gov> Co-authored-by: Alexander Winter <80929291+AlexanderWinterLLNL@users.noreply.github.com>

ben-bay and others added 3 commits June 11, 2020 17:47

sample generation is now logged to merlin_info

2737b6f

Added to CHANGELOG

aaed772

fixed style

80fc5b5

ben-bay requested review from ymubarka, koning and lucpeterson and removed request for ymubarka June 11, 2020 23:52

koning approved these changes Jun 15, 2020

View reviewed changes

merlin/study/study.py Show resolved Hide resolved

ben-bay merged commit e913381 into develop Jun 16, 2020

ben-bay deleted the feature/ben/samples_log branch June 16, 2020 17:20

ben-bay mentioned this pull request Jun 18, 2020

[FEAT] merlin block errors and outputs #117

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Log sample generation to merlin_info #234

Log sample generation to merlin_info #234

ben-bay commented Jun 11, 2020 •

edited

Loading

Log sample generation to merlin_info #234

Log sample generation to merlin_info #234

Conversation

ben-bay commented Jun 11, 2020 • edited Loading

ben-bay commented Jun 11, 2020 •

edited

Loading