bugfix/output-path-substitution #430

bgunnar5 · 2023-07-10T16:25:01Z

A user named Shailaja brought a bug to my attention that involved CLI substitutions for variables used in the OUTPUT_PATH variable. For example, say you have an environment block like so:

env:
    variables:
        DATE: temp
        OUTPUT_PATH: study_$(DATE)

If you run this study with no CLI substitutions, the output path is set as intended (i.e. it's named study_temp); however, if you try to set the DATE variable from the CLI with merlin run <spec file> --vars DATE="date +%Y%m%d-%H%M%S", the OUTPUT_PATH variable is still named study_temp which is not expected behavior.

This PR fixes that issue.

koning · 2023-07-10T17:17:53Z

is there a test this can be added to?

bgunnar5 · 2023-07-10T17:19:24Z

@koning good call. I'll check and add one

koning · 2023-07-10T17:20:38Z

OK, otherwise , looks good to me.

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov>

* Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>

* fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * remove github text that was causing errors * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * docs/conversion-to-mkdocs (#460) * remove a merge conflict statement that was missed * add base requirements for mkdocs * set up configuration for API docs * start work on porting user guide to mkdocs * add custom styling and contact page * begin work on porting tutorial to mkdocs * add new examples page * move old sphinx docs to their own folder (*delete later*) * modify some admonitions to be success * modify hello examples page and port step 3 of tutorial to mkdocs * fix typo in hello example * finish porting step 4 of tutorial to mkdocs * port part 5 of the tutorial to mkdocs * copy faq and contributing from old docs * port step 6 of tutorial to mkdocs * remove unused prereq * port step 7 of tutorial to mkdocs * add more detailed instructions on contributing * move venv page into installation and add spack instructions too * add configuration docs * add content to user guide landing page * port celery page to mkdocs * rearrange configuration pages to add in merlin server configuration instructions * port command line page to mkdocs * finish new landing page * change size of merlin logo * port variables page to mkdocs * fix broken links to configuration page * port FAQ to mkdocs * fix incorrect requirement name * update CHANGELOG * attempt to get docs to build through readthedocs * port docker page to mkdocs * port contributing guide to mkdocs * add new 'running studies' page * add path changes to images * add a page on how to interpret study output * add page on the spec file * remove old sphinx docs that are no longer needed * added README to docs and updated CHANGELOG * fix copyright and hello_samples tree * rearrange images/stylesheets and statements that use them * add suggestions from Luc and Joe * add tcsh instructions for venv activation * add Charle's suggestions for the landing page * change tcsh mentions to csh * openfoam tutorial modifications (#463) * feature/revamped status (#464) * feature/new-status (#442) * add backend functionality for merlin status * add frontend functionality for merlin status * add tests for merlin status * run fix-style and remove import of deprecated function * update CHANGELOG * add more logging statements, make better use of glob * run fix-style * clean up test files a bit * fix test suite after step_name_map mod * add avg/std dev run time calculations to status * modify status tests to accommodate new avg/std dev calculations * fix linter issues * fix lint issue and add test for avg/std dev calc * feature/detailed-status (#451) * Version/1.11.0 (#449) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) * bugfix/sphinx-5.3.0-requirement (#446) * Version/1.10.3 (#445) * fix default worker bug with all steps * version bump and requirements fix * Bugfix/filename-special-vars (#425) * fix file naming bug * fix filename bug with variable as study name * add tests for the file name special vars changes * modify changelog * implement Luc's suggestions * remove replace line * Create dependabot-changelog-updater.yml * testing outputs of modifying changelog * delete dependabot-changelog-updater * feature/pdf-docs (#427) * first attempt at adding pdf * fixing build error * modify changelog to show docs changes * fix errors Luc found in the build logs * trying out removal of latex * reverting latex changes back * uncommenting the latex_elements settings * adding epub to see if latex will build * adding a latex engine variable to conf * fix naming error with latex_engine * attempting to add a logo to the pdf build * testing an override to the searchtools file * revert back to not using searchtools override * update changelog * bugfix/openfoam_singularity_issues (#426) * fix openfoam_singularity issues * update requirements and descriptions for openfoam examples * bugfix/output-path-substitution (#430) * fix bug with output_path and variable substitution * add tests for cli substitutions * bugfix/scheduler-permission-error (#436) * Release/1.10.2 (#437) * bump version to 1.10.2 * bump version in CHANGELOG * resolve develop to main merge issues (#439) * fix default worker bug with all steps * version bump and requirements fix * dependabot/certifi-requests-pygments (#441) * Bump certifi from 2022.12.7 to 2023.7.22 in /docs Bumps [certifi](https://github.com/certifi/python-certifi) from 2022.12.7 to 2023.7.22. - [Commits](certifi/python-certifi@2022.12.07...2023.07.22) --- updated-dependencies: - dependency-name: certifi dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> * add all dependabot changes and update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * bugfix/server-pip-redis-conf (#443) * add *.conf to the MANIFEST file so pip will grab the redis.conf file * add note explaining how to fix a hanging merlin server start * modify CHANGELOG * add second export option to docs and fix typo * bump to version 1.10.3 (#444) --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * change hardcoded sphinx requirement * update CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * feature/vlauncher (#447) * fix file naming error for iterative workflows * fixed small bug with new filepath naming * add VLAUNCHER functionality * add docs for VLAUNCHER and modify changelog * re-word docs and fix table format * add a test for vlauncher * run fix-style and add a test for vlauncher * Add the find_vlaunch_var and setup_vlaunch functions. The numeric value of the shell variables may not be defined until run time, so replace with variable strings instead of values. Consolidate the commands into one function. * Add variable set for (t)csh. * Run fix-style * make step settings the defaults and ignore commented lines * add some additional tests * remove regex library import --------- Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * release/1.11.0 (#448) * bugfix/skewed-sample-hierarchy (#450) * add patch for skewed sample hierarchy/additional samples * update changelog * catch narrower range of exceptions --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> * add functionality for the detailed-status command * add tests for detailed-status * fix linter issues * update changelog * general cleanup and add log statements * slightly modify two tests * default status renderer now uses json status format * remove inaccurate comment * bugfix/lsf-gpu-typo (#453) * fix typo in batch.py that causes a bug * change print statements to log statements * release/1.11.1 (#454) * Add Pytest Fixtures to Test Suite (#456) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * Bugfix for WEAVE CI (#457) * begin work on integration refactor; create fixtures and initial tests * update CHANGELOG and run fix-style * add pytest fixtures and README explaining them * add tests to demonstrate how to use the fixtures * move/rename some files and modify integration's README * add password change to redis.pass file * fix lint issues * modify redis pwd for test server to be constant for each test * fix lint issue only caught on github ci * add fix for merlin server startup * update CHANGELOG * bugfix/monitor-shutdown (#452) * add celery query to see if workers still processing tasks * fix merlin status when using redis as broker * fix consumer count bug and run fix-style * fix linter issues * update changelog * update docs for monitor * remove unused exception I previously added * first attempt at using pytest fixtures for monitor tests * (partially) fix launch_workers fixture so it can be used in multiple classes * fix linter issues and typo on pytest decorator * update black's python version and fix style issue * remove print statements from celeryadapter.py * workers manager is now allowed to be used as a context manager * add one thing to changelog and remove print statement * Add the missing restart keyword to the specification docs. (#459) * add Jeremy's suggestion to change vars option to output-path * remove unnecessary lines from CHANGELOG --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * feature/queue info (#461) * remove a merge conflict statement that was missed * add queue-info functionality * add tests for queue-info * update CHANGELOG * add try/except for forceful termination of test workers * change github workflow to use py38 with black instead of py36 * run fix-style with py 3.12 and fix a typo in a test * add filetype check for dump option * add banner print statement * docs/revamped status (#462) * fix broken image link in README * add new commands to the command line page * add monitoring docs layout and complete status cmds page * fix bug with dumping queue-info to files * add docs for queue-info * add documentation for 'query-workers' * add reference to new query-workers docs and split a paragraph * fix small bug with --steps option of monitor * add documentation for monitor command * update CHANGELOG * fix dump-csv image for queue-info --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> * release/1.12.0 (#465) * remove a merge conflict statement that was missed * bump version to 1.12.0 * feature/retry_priority (#468) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * add new retry priority as highest task priority * update CHANGELOG * add in MID priority * change default priority to use priority map MID value * docs/server-cross-node (#470) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * rename the merlin server config page * add instructions for running a cross-node workflow w/ containerized server * update CHANGELOG * bugfix/initial-status-issues (#471) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * fix bug with dry run status * set MANPAGER for detailed-status * fix bug with 1 sample removing the status file * add support for multiple workers on one step in status files * update test suite to accommodate changes to workers in status files * add catch and potential fix for JSONDecodeError * fix docstring of a test * update CHANGELOG.md * run fix style and add Luc's suggestions * run fix-style with python 3.12 * added additional check for status file while condensing * add try/except to catch an error for dumping statuses * release/1.12.1 (#472) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * bump version to 1.12.1 * fix a lint issue that somehow slipped through the cracks * Fix filenames for OpenFoam tutorial (#475) * bugfix/deep-merge-existing-keys (#476) * remove a merge conflict statement that was missed * add a 'pip freeze' call in github workflow to view reqs versions * remove DeepMergeException and add conflict_handler to dict_deep_merge * add conflict handler to dict_deep_merge * fix broken tests for detailed-status * use caplog fixture rather than IO stream * add ability to define module-specific fixtures * add tests for read/write status files and conlict handling * add caplog explanation to docstrings * update CHANGELOG * run fix-style * add pytest-mock as dependency for test suite * clean up input check in dict_deep_merge * Improved Info (#477) * Add merlin version to banner * Add python package info to and clean up 'merlin info' * Add some unit tests * Force GitHub runner checkout to grab the whole history, fixing CHANGELOG test bug * Update CHANGELOG to show bugfix to CHANGELOG test * Target is in source's history (#478) * New github action test to make sure target has been merged into source * Fix link to merlin banner image (#479) * bugfix/status_nested_workspace (#480) * remove a merge conflict statement that was missed * have status ignore nested workspaces and modify merge rules * update CHANGELOG * fixed issue with escape sequences in ascii art * apply Luc's suggestion * add setuptools as a requirement since python 3.12 doesn't have it natively * modify unit tests for status to use pytest rather than unittest * update CHANGELOG * add fixtures for status testing and add nested workflow test * update CHANGELOG * bugfix/celery-chord-error (#481) * remove a merge conflict statement that was missed * add celery results backend patch to stop ChordErrors * add MERLIN_RAISE_ERROR return code * add tests to ensure chord error isn't raised * add RAISE_ERROR to docs * update CHANGELOG * fix lint issues * up the sleep time on the chord error test * add new steps to the chord err test spec * add tree statement to the new test for debugging * upping sleep time to see if that fixes github action for python 3.7 * change sleep time for new test based on python version * run fix style * remove specific sleep time for diff python versions * release/1.12.2b1 (#482) * remove a merge conflict statement that was missed * bump version to 1.12.2b1 * bugfix/flux-nodes (#484) * remove a merge conflict statement that was missed * fix flux node allocation issue * allow for vars to be used with nodes settings of workers/batch * add tests for var usage with nodes * update CHANGELOG * run fix-style * bugfix/flux-nodes-prior-versions (#487) * add a version check for flux when getting node count * update CHANGELOG * add major version check for flux * Change Task ID to directory path (#486) * Modifying task id to include directory * Adding Several New Unit Tests (#490) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * split up create_server_config and write tests for it * add tests for config_merlin_server function * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * update CHANGELOG * add Ryan and Joe's suggestions * update tests to use newly named functions * fix linter issue * release/1.12.2 (#491) * update version to 1.12.2 * fix style issue --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Jane Herriman <xorJane@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov>

fix bug with output_path and variable substitution

a969f33

bgunnar5 requested review from koning, lucpeterson and doutriaux1 July 10, 2023 16:25

add tests for cli substitutions

2236629

koning approved these changes Jul 10, 2023

View reviewed changes

bgunnar5 merged commit 12ff3d7 into LLNL:develop Jul 11, 2023

bgunnar5 deleted the bugfix/output-path-substitution branch July 11, 2023 18:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

bugfix/output-path-substitution #430

bugfix/output-path-substitution #430

bgunnar5 commented Jul 10, 2023

koning commented Jul 10, 2023

bgunnar5 commented Jul 10, 2023

koning commented Jul 10, 2023

bugfix/output-path-substitution #430

bugfix/output-path-substitution #430

Conversation

bgunnar5 commented Jul 10, 2023

koning commented Jul 10, 2023

bgunnar5 commented Jul 10, 2023

koning commented Jul 10, 2023