Bugfix/luc/shutdown task acknowledgement #182
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Solid. Much better than my previous hack. |
ben-bay
approved these changes
Mar 12, 2020
|
I get this output: It may cause confusion that is is an error. I believe this is coming from maestro. |
|
You're right. This is coming from the maestro localscriptadapter's submit function It actually pops up for anything that returns a non-success code, so we should address this with our own submit function |
|
@koning see the latest change, which pulls out the maestro submission, giving us more control. this also removes that "error" message |
ben-bay
reviewed
Mar 13, 2020
|
Looks good, I'll create maestro issue for this. I am going to suggest that maestro look in a list of return codes for OK. |
…ewer file system calls.
…ting down everything
koning
reviewed
Mar 13, 2020
| run: | ||
| cmd: | | ||
| exit $(MERLIN_STOP_WORKERS) | ||
| exit $(MERLIN_HARD_FAIL) |
There was a problem hiding this comment.
Why do you want to shutdown only some of the workers?
There was a problem hiding this comment.
hard_fail will stop all the workers in that workflow, since they all look at that queue; stop_workers stops all workers everywhere. i've put a note in there
There was a problem hiding this comment.
Ok, but in general HARD_FAIL will only stop the workers for queues associated with the specific failed step. Is this correct?
There was a problem hiding this comment.
as an alternative, it could shutdown all workers associated with all the queues in that workflow, but that would involve getting information that lives outside of the individual step. it might also have the side effect of bleeding into other workflows' workers via the default queue.
right now it's easiest to just stop workers on that specific queue, which i think should encourage users to be conscious of which queue they want to shut down
…IL vs STOP_WORKERS
koning
approved these changes
Mar 13, 2020
This was referenced Mar 13, 2020
ben-bay
added a commit
that referenced
this pull request
Mar 27, 2020
* removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 * Allow for empty username when using redis. (#163) * Update CHANGELOG.md * corner case bugfix (#164) * fixed bug * updated CHANGELOG * Change hello world sample generator to be much much faster (#165) * Add the rediss (ssl_redis) server and ssl keys in app.yaml. (#170) * Add the rediss server. Allow users to define ssl keys either as full paths or in the celery/certs dir. Add a url option to override the rest of the app.yaml broker or results backend config. Add function docstrings. * Add default results_ssl variable. * Fix typos. * Add missing docker-compose TLS file. (#172) * Add batch nodes = "all" option. (#174) * Update CHANGELOG.md * merlinwf -> merlin (#168) * merlinwf -> merlin * py-merlinwf -> py-merlin * removed 2 final instances of merlinwf * Feature/luc/zfill dirs (#176) * Fix uniform_directories to include the bundle size, Zero pad the directories to make sequential Remove the extra layer in the directory hierarchy (which was always 0) * Update CHANGELOG * fixed paths (#180) * Bugfix/luc/shutdown task acknowledgement (#182) * Add MERLIN_STOP_WORKERS exit code * Add shutdown to feature_demo.yaml; this should fix dsitributed integration test * MerlinScriptAdapter uses its own submission function * Modify MERLIN_HARD_FAIL to shutdown only the workers connected to that step * Add a comment in feature demo about MERLIN_HARD_FAIL vs MERLIN_STOP_WORKERS * Remove the sleep inf for flux. (#184) Add a monitor command which will block until no celery workers are running, polling every 60 seconds. Add docs. Alphabetize the merlin commands. Fix typos and add workers stopped info for monitor. Add arguments for merlin monitor. Convert the sbatch and bsub batch scripts to use merlin monitor. Add exit $(MERLIN_STOP_WORKERS) for the merlin monitor completion. * typo fixes + doc tweaks (#185) * docs tweaks * fixed typos * fixed typo * tweak * Tutorials -> dev (#186) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code * v1.4.1 (#162) * removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * added figures and updated document…
ben-bay
added a commit
that referenced
this pull request
Mar 31, 2020
* removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 * Allow for empty username when using redis. (#163) * Update CHANGELOG.md * corner case bugfix (#164) * fixed bug * updated CHANGELOG * Change hello world sample generator to be much much faster (#165) * Add the rediss (ssl_redis) server and ssl keys in app.yaml. (#170) * Add the rediss server. Allow users to define ssl keys either as full paths or in the celery/certs dir. Add a url option to override the rest of the app.yaml broker or results backend config. Add function docstrings. * Add default results_ssl variable. * Fix typos. * Add missing docker-compose TLS file. (#172) * Add batch nodes = "all" option. (#174) * Update CHANGELOG.md * merlinwf -> merlin (#168) * merlinwf -> merlin * py-merlinwf -> py-merlin * removed 2 final instances of merlinwf * Feature/luc/zfill dirs (#176) * Fix uniform_directories to include the bundle size, Zero pad the directories to make sequential Remove the extra layer in the directory hierarchy (which was always 0) * Update CHANGELOG * fixed paths (#180) * Bugfix/luc/shutdown task acknowledgement (#182) * Add MERLIN_STOP_WORKERS exit code * Add shutdown to feature_demo.yaml; this should fix dsitributed integration test * MerlinScriptAdapter uses its own submission function * Modify MERLIN_HARD_FAIL to shutdown only the workers connected to that step * Add a comment in feature demo about MERLIN_HARD_FAIL vs MERLIN_STOP_WORKERS * Remove the sleep inf for flux. (#184) Add a monitor command which will block until no celery workers are running, polling every 60 seconds. Add docs. Alphabetize the merlin commands. Fix typos and add workers stopped info for monitor. Add arguments for merlin monitor. Convert the sbatch and bsub batch scripts to use merlin monitor. Add exit $(MERLIN_STOP_WORKERS) for the merlin monitor completion. * typo fixes + doc tweaks (#185) * docs tweaks * fixed typos * fixed typo * tweak * Tutorials -> dev (#186) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code * v1.4.1 (#162) * removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * added figures and updated documentation * …
ben-bay
added a commit
that referenced
this pull request
May 7, 2020
* v1.5.1 (#200) * removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 * Allow for empty username when using redis. (#163) * Update CHANGELOG.md * corner case bugfix (#164) * fixed bug * updated CHANGELOG * Change hello world sample generator to be much much faster (#165) * Add the rediss (ssl_redis) server and ssl keys in app.yaml. (#170) * Add the rediss server. Allow users to define ssl keys either as full paths or in the celery/certs dir. Add a url option to override the rest of the app.yaml broker or results backend config. Add function docstrings. * Add default results_ssl variable. * Fix typos. * Add missing docker-compose TLS file. (#172) * Add batch nodes = "all" option. (#174) * Update CHANGELOG.md * merlinwf -> merlin (#168) * merlinwf -> merlin * py-merlinwf -> py-merlin * removed 2 final instances of merlinwf * Feature/luc/zfill dirs (#176) * Fix uniform_directories to include the bundle size, Zero pad the directories to make sequential Remove the extra layer in the directory hierarchy (which was always 0) * Update CHANGELOG * fixed paths (#180) * Bugfix/luc/shutdown task acknowledgement (#182) * Add MERLIN_STOP_WORKERS exit code * Add shutdown to feature_demo.yaml; this should fix dsitributed integration test * MerlinScriptAdapter uses its own submission function * Modify MERLIN_HARD_FAIL to shutdown only the workers connected to that step * Add a comment in feature demo about MERLIN_HARD_FAIL vs MERLIN_STOP_WORKERS * Remove the sleep inf for flux. (#184) Add a monitor command which will block until no celery workers are running, polling every 60 seconds. Add docs. Alphabetize the merlin commands. Fix typos and add workers stopped info for monitor. Add arguments for merlin monitor. Convert the sbatch and bsub batch scripts to use merlin monitor. Add exit $(MERLIN_STOP_WORKERS) for the merlin monitor completion. * typo fixes + doc tweaks (#185) * docs tweaks * fixed typos * fixed typo * tweak * Tutorials -> dev (#186) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code * v1.4.1 (#162) * removed conflict artefact * Tutorials -> develop (#161) * Made skeleton of tutorial modules * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * added pip and docker sections to outline * made tutorial modules numbered * added rough headings to installation * Tutorial/koning Installation (#120) * Add the initial installation tutorial step. * Fix bullets and typos. * Fix redis server edit. * Fix typo. * Add --detach in doc and fix typo in docker run doc. * Fix prompt. * Add upgrade of setuptools pip. Add make test to local rdis. Fix typo in redis-server path. * Add python3 and fix celery alias. * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * adding to module headers * upgraded header formatting * tutorial updates * working on hello world * adding to hello world * updated hello world * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * adding log files to hello world * updates * continued outline * Docs/koning/installation (#123) * Remove check and add server check to info. * Add app.yaml examples and rabbit server info. Add server connection check to merlin info. * Update with some rabbitmq info and the server check in merlin info. * Add the -p docker run option. * Fix more typos. * Run fix-style. * Add check server access to CHANGELOG. * Fix sphinx bugs. * Add admonition and C++ compiler. * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 * Update installation with docker-compose sections. Remove pip mysql install from deafult make. Update spec with new info. * hello world -> tutorials (#140) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * Add newline. * added prereqs * Fix indentation. * Fix info section. * Indentation in the merlin info output. * Hotfix/ben/doc error (#137) * develop -> master (#136) * Add an initial Dockerfile with an entrypoint running merlin by default. (#110) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Update CHANGELOG.md * Revert "Add an initial Dockerfile with an entrypoint running merlin by default. (#110)" This reverts commit 95985b80d35f6c2ad8724cdea21dacdfedb78871. * web docs updates (#111) * updating Sphinx Makefile * Updated web doc theme to alabaster. * updating outdated sphinx documentation * Update spack docs. * Fix wraps and co->checkout. * added description, GitHub link * Remove extra char. * hid tree in the body, made sidebar fixed * added --system-site-packages Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Remove unneeded target * Update dev.txt * updated Makefile (#113) * updated Makefile * release target runs python3 * Feature/ben/makefile (#116) * updated Makefile * release target runs python3 * updated Makefile * removed git-related targets * Update CHANGELOG.md * cli test simplification (#106) * added --local flag to cli tests, to remove the need to list out all 24 ids when testing merlin locally. * commented out one local tag * removed comment * updated CHANGELOG * fixed local cli tests * added twine to dev deps * Add a Dockerfile (#115) * Add an initial Dockerfile with an entrypoint running merlin by default. * Add docker docs. * Add docker to index. * Add cffi explicitly. * Add a user, merlinu, to the Dockerfile and switch to /home/merlinu for the workdir. Update the docs with the new volume. * Add dev and feature_demo requirements. * Force python3 to be python3.8. Add python2 for feature_demo. Change all python executables to python3. * Update Dockerfile for python3.8 and add python2. * Add comment about change from python to python3. * Fix typos. * Fix typos and move to feature_demo as example. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add the --broker keyword to merlin config and config parse fixes (#114) * Add the -broker keyword to merlin config so a pure redis config can be used instead of the rabbitmq. merlin config --broker redis default is rabbitmq Add a redis app.yaml file. Remove the requirement for celery keyword in the app.yaml config file. * Update CHANGELOG.md * Run fix-style. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * target a spec with stop-workers (#118) * updating stop workers logic * updates * fixed style * corrected debug print * added debug print * added to condition * added debug print * added debug print * debugging stop-workers with --spec * fixed style * added to documentation * added link from faq to merlin_commands * removed unused indices and tables * removed venv, since pip is the new standard install * expanded a menu title * Update docs (#119) * Update docs with actual merlin commands in paragraph titles. Add --broker to config. * Add missing options to run and run-workers. Change heading for run-workers. * Fix typos and move --vars Example to rest code. * Remove celery for more generality. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * doc improvements (#121) * corrected required sphinx version * updated Makefile * fixed external link * documentation improvements * removed numbering * adding $ to shell commands * added next/previous buttons to top and bottom * more updates * removed incorrect html option * removed unneeded variables from simple_chain.yaml (#122) * removed unneeded variables from simple_chain.yaml * added to CHANGELOG * changed python and pip to v3 in merlin info (#125) * changed python and pip to v3 in merlin info * fixed style * updated CHANGELOG * removed references to celery in non-debug log messages (#128) * removed all info-level logger references to Celery. * fixed style * docker updates (#130) * Change to recursive-include in MANIFEST.in Add author and author_email to isetup.py. Fix the user in Dockerfile and remove entrypoint due to the need for both merlin and celery from the container. * Add the docker linking docs. * Make docs more readable. * More readability. * Add --broker doc. * Fix docs remove -i, change llnl/redis to llnl/merlin. * Allow the port to be changed in rabbitmq broker. * Alter rabbitmq output strings to match the others. * Change rabbitmq default port output from warning to debug. * Add changes. * Make CHANGELOG location changes. * Update CHANGELOG.md * command line docs (#134) * adding merlin status section * updating merlin commands;2C * updated CHANGELOG * added docs on --vars to purge command * added explanation for --task-server * fixed typo * added hyphen * Add a check server access when `merlin info` is run. (#135) * Add a check server access when `merlin info` is run. and a port option to rabbitmq config options. * Change from broker and backend to broker server and results server. * Add minimal black changes. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * incremented version to v1.2.4 * added date to CHANGELOG * incremented to 1.3.0 Co-authored-by: Joe Koning <koning@users.noreply.github.com> * fixed sphinx syntax errors * incremented version, updated CHANGELOG * reverted premature version increment Co-authored-by: Joe Koning <koning@users.noreply.github.com> * Rewrite docker redis instead of local-redis. * Initial commit of advanced section. (#139) * Docs/ben/contributing (#142) * added material to contributing.rst * debugged contribte.rst * hello world (#143) * progress * added hello world material * updates * changes * adding hello improvements * updates * faker samples * hello_samples is functioning * switched from french to portuguese * switched from faker to names * adjusted workflows * added make samples script * made working make_samples.py * updates * tutorial now appears to be working * changed section name * Enable walltime in spec (#138) * Add an override get_parallelize_command to be able to use keys with an equal sign. * Add walltime for slurm and flux LAUNCHER, lsf does not have a time arg for the parallel launch. Add a signal option for slurm LAUNCHER. * Update CHANGELOG.md. * Runn black on one file. * Update spec with new run LAUNCHER args. * Add lsf, slurm, flux keywords for run section to add launcher specific keywords. Update FAQ. * Add close paren in FAQ. * Change wording for LAUNCHER params. * Fix typos. * Make parallel form for LAUNCHER specific flags. * Run black on script adapter. * various hello world improvements * added sample file descriptions * added emojis * added images to tutorial * removed old image * added hello workflows to internal examples, for easier copying * Added files * Intro outline. Some fleshing out of first subsection (#127) * Intro outline. Some fleshing out of first subsection * Add images and some discussion * Figure mods, some reordering * Nearly done with intro 1.1 * Rejiggering of sections. Getting pretty close * Pushing into final subsection * Done drafting intro * Adding dynamic sampling workflow (#144) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Add rabbitmq environment variables to get the rabbit server to work. * Add docker-compose scripts and remove installation scripts. * Fix VERIFY and typo in PEER. * eliminated duplicate scripts * FIx header in merlin_config Add app.yaml and docker-compose script as literalincludes. * Fix wording in docker redis note. * Change docker to required for module 4. * Add rabbit password. * Change filename to rabbit.pass. * Openfoam -> tutorials (#148) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * Run black on make_samples.py * Openfoam -> tutorials (#149) * changes * editing the learning step and added documentation * added more documentation and matched the DAG to discription * added run simulation documentation * fixed indentation * sim doc updates * fixed sphinx contents bug * fixed upper links * removed contents * fixed some links * clean up some sections * added more documentation * improvements * finished putting up the spec * moved openfoam to examples * file now correctly displays * altered heading * removed duplicate doc * shifted sections * Added documentation, split up the last step * added dag * removed misbehaving tests * updatd image * sorted sections * reduced redundency and added documentation * added prediction graphs * added image Co-authored-by: Yamen Mubarka <mubarka1@llnl.gov> * added to contrib * Feature/koning/display exceptions (#150) * Print out exceptions when running merlin info, if any are present, without stopping. * Record all connection exceptions. * Run black on display.py * added to changelog, fixed style * incremented to v1.4.0 * added to changelog * added tester message * added tester message * removed stop * Remove exclusive and signal in slurm in favor of the slurm keyword. (#152) * Remove exclusive and signal in slurm in favor of the slurm keyword. Make bind lsf only. * Fix typo. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * Advanced tutorial example cleanup (#153) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * amended py req * Fix missing directory error in dynamic sampling demo scripts (#154) * Initial commit of advanced section. * Fix reserved variable table. * Add dynamic sampling workflow scripts for hello world variant. * Move advanced files to subdir, add additional post processing script. * Add dynamic sampling workflow to docs. * Threaded post proc scripts, cleaned up some script docstrings. * Add requirements file for advanced virtual environment. * Add required python pacckages. * Fix up some variable name typos, indentation issues. * Add iterative workflow results. * Update dynamic results. * Add check to create directories if they don't exist. Co-authored-by: Benjamin Bay <48391872+ben-bay@users.noreply.github.com> * changes * moved rst files into their respective dirs * removed comment * fixed sphinx build errors * added empty template, updated run_simulation module * updated links * added table of contents * Made the before section start at 0 * syntax changes * added a description block * fixed plot title * added a black edge to the dots * added a copy button for code blocks * updates * hello improvements * Tut/koning/installation update (#155) * Make changes to installation readability and details. * Remove extra comma. * updates * added inputs vs outputs and fixed annotation * no change * re-added hello.yaml * updated graph tables, removed merlin from reqs, updated module 4 * Add a comment on shutting down the redis server. * Change results to result. * Some typo changes, adding links (#157) * updates * temporary commit * fixed style * added stop-workers to hello tutorial * small addition * added to CHANGELOG * updated llnl code Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov> * incremented to v1.4.1 Co-authored-by: Joe Koning <koning@users.noreply.github.com> Co-authored-by: Joseph M. Koning <koning1@llnl.gov> Co-authored-by: Jeremy White <44277022+jwhite242@users.noreply.github.com> Co-authored-by: Mubarka, Yamen <mubarka1@llnl.gov> Co-authored-by: Luc Peterson <peterson76@llnl.gov>…
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a return code to allow for shutdown of workers.
Does this via a special task that is launched in a delay, to allow the parent task to finish. The custom task is acknowledge as soon as it's pulled from the server, preventing re-queueing loop.