Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

required executables #195

Closed
nsheff opened this issue Jan 21, 2017 · 11 comments
Closed

required executables #195

nsheff opened this issue Jan 21, 2017 · 11 comments
Assignees
@vreuter
Labels
likely-solved
Milestone
1.5.0

Comments

@nsheff
Copy link
Contributor

nsheff commented Jan 21, 2017

It would be nice if a pipeline could specify what executables it needs.

This could happen in the pipeline_interface -- a new attribute called required_executables, and perhaps a second called required_config_executables, could be read by looper, which could test these executables to make sure they are in the PATH.
@afrendeiro
Copy link
Contributor

There was some groundwork for this in the CommandChecker class. I guess it could be adapted/integrated in that.

@vreuter
Copy link
Member

vreuter commented Mar 21, 2017

Feel free to de-escalate if you wish @nsheff but this seems valuable...

@vreuter
Copy link
Member

vreuter commented Mar 21, 2017 •
edited
Loading

Could we infer this? Something like:
1 -- determine pipelines that should be run and from which pipelines location
2 -- grab config file for each such pipeline (from the relevant location)
3 -- parse the tools and resources sections from each such config
4 -- determine which pipeline(s)--if any--are missing necessary environment variable/path settings and provide such feedback, either proceeding with the pipelines that are OK, or failing altogether

@nsheff
Copy link
Contributor Author

nsheff commented May 17, 2017

This could lead to a new command like looper executable that would test pipelines for if they will run or not...

@vreuter
Copy link
Member

vreuter commented May 17, 2017

I like that idea. What about execs? Or making check be flags and letting this concept be check?

@nsheff
Copy link
Contributor Author

nsheff commented May 17, 2017

I like both suggestions.

@vreuter vreuter self-assigned this May 29, 2017
@nsheff
Copy link
Contributor Author

nsheff commented Oct 19, 2017

I don't think it would be possible to infer this. I don't really want to impose structure on the config file here, and it's not just config settings that would determine executables.

I think the only way to really do it well is for the pipeline author to specify a list, and the best place for that is in the pipeline interface.

@vreuter
Copy link
Member

vreuter commented Oct 19, 2017

I agree; should we implement the pipeline interface specification way for this version?

@nsheff
Copy link
Contributor Author

nsheff commented Oct 19, 2017

if it's not too much work. otherwise we can wait for 0.8

@nsheff nsheff transferred this issue from pepkit/peppy May 29, 2019
@vreuter vreuter mentioned this issue Jun 21, 2019
Open
vreuter added a commit that referenced this issue Jun 22, 2019
@vreuter
first pass at infrastructure for pipeline requirements declaration; #195
8aabf59 
; databio/pypiper#94
@nsheff
Copy link
Contributor Author

nsheff commented Mar 17, 2020

with #225 we've now started validating PEPs according to a schema. This issue is about making sure the pipeline can run; in a sense, validating an environment.

Does it make sense to do something similar? Define an environment schema, and then validate that?

@nsheff nsheff added this to the 0.15 milestone Mar 25, 2020
donaldcampbelljr added a commit that referenced this issue Aug 3, 2023
@donaldcampbelljr
attempt simple check to see if provided pipelines are callable #195
6513b8a
@donaldcampbelljr donaldcampbelljr mentioned this issue Aug 3, 2023
Merged
donaldcampbelljr added a commit that referenced this issue Aug 9, 2023
@donaldcampbelljr @nsheff
@khoroshevskyi @vreuter @ayobi
Looper v1.5.0 release (#388)
6ac3b12 
* add geofetch tutorial

* Add pephub support for sample-level pipeline interface.

* Fixed tests naming

* added project level for pephub support

* simplify Project constructor implementation

* logger string fix

Co-authored-by: Vince <vince.reuter@gmail.com>

* fix try block

Co-authored-by: Vince <vince.reuter@gmail.com>

* Update looper/utils.py

Co-authored-by: Vince <vince.reuter@gmail.com>

* Fixed #341 #342

* Deprecated write_skipped_sample_scripts. Scripts will now output sequentially (unless toggled). #173

* Remove redundant pooling behavior for skipped samples. #173

* First working version of #344

* divvy reintegration #343

* divvy reintegration #343

* Cleaned up looper with divvy

* removed sample and project pipelines from cli

* fixed old .looper.config specification

* added new config docs

* Remove --toggle key. Add logic during fetch samples for toggle. Remove checking for toggle off during sample submission (redundant). #263

* Remove toggle key constants. Apply formatting. #263

* Remove toggle key property. Simplified logic for fetching samples. #263

* fix pephub failing tests

* cleaned up more to help pass pytests

* distutils.dir_util for shutil for python 3.12

* added divvy entry point inside looper

* black format

* another black reformat

* added docs

* clean up based on feedback

* clean up based on feedback

* removed redundancy

* black fmt

* added divvy inspect

* black fmt

* added sub cmd, docker args

* added line break for inspect output

* added divvy docs #343

* added divvy docs #343

* divvy docs fix

* mkdocs fix

* Fixed mkdocs error

* Update requirements-doc.txt

* updated reqs-doc

* merge mistake fix

* added divvy imgs

* added new looper init

* added to changlog, fix divvy imgs

* divvy readme img fix

* fixed initialization of generic piface

* fixed initialization of generic piface

* added tests

* fixed main setup

* Update how_to_define_looper_config.md

* Update __init__.py

* Update test_other.py

* Ise

* added changelog and minor naming changes

* remove old logging function

* dev version bump

* fix typo in html_report and upgraded pandas requirements for pephubclient

* fixed requirements

* fixed docs requirements

* added versioneer to doc requirements

* added Cython to doc requirements

* added readthedocs config

* added looper to requirements docs

* allow for using pipestat.summarize, align with pipestat 0.4.0

* clean up code, update usage doc

* update doc requirements pephubclient

* downgrade docs to 3.10

* adjust get_status to use proper sample_name if pipestat configured #326

* adjust conductor to retrieve pipestat manager variables with pipestat 0.4.0 refactoring.

* Allows skipping some tests if run offline. Closes #370

* work on using test_args instead of subprocesses

* Finish switching applicable tests away from subprocess

* Lint and update doc string to test_args_expansion

* Change return type.

* lint

* add test for var_templates #357, and clean up tests

* attempt simple check to see if provided pipelines are callable #195

* minor adjustments, polished docstring

* update changelog

* lint

* update version to 1.5.0

* update changelog

* update reqs and changelog to use pipestat v0.5.0

* Refactoring for looper config

* added looper config file argument

* code fix

* Added comment about deprecating for old looper specification

* fixed looper init error

* change logo for docs build tab icon

* fix favicon

* update docs and changelog for 1.5.0 release

---------

Co-authored-by: nsheff <nsheff@users.noreply.github.com>
Co-authored-by: Khoroshevskyi <sasha99250@gmail.com>
Co-authored-by: Vince Reuter <vince.reuter@gmail.com>
Co-authored-by: ayobi <17304717+ayobi@users.noreply.github.com>
@donaldcampbelljr donaldcampbelljr mentioned this issue Aug 9, 2023
Merged
donaldcampbelljr added a commit that referenced this issue Aug 14, 2023
@donaldcampbelljr @nsheff
@khoroshevskyi @vreuter @ayobi
v1.5.1 (#391)
145c0d6 
* add geofetch tutorial

* Add pephub support for sample-level pipeline interface.

* Fixed tests naming

* added project level for pephub support

* simplify Project constructor implementation

* logger string fix

Co-authored-by: Vince <vince.reuter@gmail.com>

* fix try block

Co-authored-by: Vince <vince.reuter@gmail.com>

* Update looper/utils.py

Co-authored-by: Vince <vince.reuter@gmail.com>

* Fixed #341 #342

* Deprecated write_skipped_sample_scripts. Scripts will now output sequentially (unless toggled). #173

* Remove redundant pooling behavior for skipped samples. #173

* First working version of #344

* divvy reintegration #343

* divvy reintegration #343

* Cleaned up looper with divvy

* removed sample and project pipelines from cli

* fixed old .looper.config specification

* added new config docs

* Remove --toggle key. Add logic during fetch samples for toggle. Remove checking for toggle off during sample submission (redundant). #263

* Remove toggle key constants. Apply formatting. #263

* Remove toggle key property. Simplified logic for fetching samples. #263

* fix pephub failing tests

* cleaned up more to help pass pytests

* distutils.dir_util for shutil for python 3.12

* added divvy entry point inside looper

* black format

* another black reformat

* added docs

* clean up based on feedback

* clean up based on feedback

* removed redundancy

* black fmt

* added divvy inspect

* black fmt

* added sub cmd, docker args

* added line break for inspect output

* added divvy docs #343

* added divvy docs #343

* divvy docs fix

* mkdocs fix

* Fixed mkdocs error

* Update requirements-doc.txt

* updated reqs-doc

* merge mistake fix

* added divvy imgs

* added new looper init

* added to changlog, fix divvy imgs

* divvy readme img fix

* fixed initialization of generic piface

* fixed initialization of generic piface

* added tests

* fixed main setup

* Update how_to_define_looper_config.md

* Update __init__.py

* Update test_other.py

* Ise

* added changelog and minor naming changes

* remove old logging function

* dev version bump

* fix typo in html_report and upgraded pandas requirements for pephubclient

* fixed requirements

* fixed docs requirements

* added versioneer to doc requirements

* added Cython to doc requirements

* added readthedocs config

* added looper to requirements docs

* allow for using pipestat.summarize, align with pipestat 0.4.0

* clean up code, update usage doc

* update doc requirements pephubclient

* downgrade docs to 3.10

* adjust get_status to use proper sample_name if pipestat configured #326

* adjust conductor to retrieve pipestat manager variables with pipestat 0.4.0 refactoring.

* Allows skipping some tests if run offline. Closes #370

* work on using test_args instead of subprocesses

* Finish switching applicable tests away from subprocess

* Lint and update doc string to test_args_expansion

* Change return type.

* lint

* add test for var_templates #357, and clean up tests

* attempt simple check to see if provided pipelines are callable #195

* minor adjustments, polished docstring

* update changelog

* lint

* update version to 1.5.0

* update changelog

* update reqs and changelog to use pipestat v0.5.0

* Refactoring for looper config

* added looper config file argument

* code fix

* Added comment about deprecating for old looper specification

* fixed looper init error

* change logo for docs build tab icon

* fix favicon

* update docs and changelog for 1.5.0 release

* - fix `looper table` failing without sample.protocol , update change log for point release

* fix "--looper-config"

* update version and changelog date

* adjust pipestat requirement to be >=0.5.1

* lint

---------

Co-authored-by: nsheff <nsheff@users.noreply.github.com>
Co-authored-by: Khoroshevskyi <sasha99250@gmail.com>
Co-authored-by: Vince Reuter <vince.reuter@gmail.com>
Co-authored-by: ayobi <17304717+ayobi@users.noreply.github.com>
@donaldcampbelljr donaldcampbelljr mentioned this issue Oct 6, 2023
Closed
@donaldcampbelljr
Copy link
Contributor

some thoughts:
write a schema to define executables (JSON schema format), write a validation program to validate the computing environments.

donaldcampbelljr added a commit that referenced this issue Dec 22, 2023
@nsheff @khoroshevskyi
@donaldcampbelljr @ayobi
Release 1.6.0 (#400)
b47a568 
* added new config docs

* Remove --toggle key. Add logic during fetch samples for toggle. Remove checking for toggle off during sample submission (redundant). #263

* Remove toggle key constants. Apply formatting. #263

* Remove toggle key property. Simplified logic for fetching samples. #263

* fix pephub failing tests

* cleaned up more to help pass pytests

* distutils.dir_util for shutil for python 3.12

* added divvy entry point inside looper

* black format

* another black reformat

* added docs

* clean up based on feedback

* clean up based on feedback

* removed redundancy

* black fmt

* added divvy inspect

* black fmt

* added sub cmd, docker args

* added line break for inspect output

* added divvy docs #343

* added divvy docs #343

* divvy docs fix

* mkdocs fix

* Fixed mkdocs error

* Update requirements-doc.txt

* updated reqs-doc

* merge mistake fix

* added divvy imgs

* added new looper init

* added to changlog, fix divvy imgs

* divvy readme img fix

* fixed initialization of generic piface

* fixed initialization of generic piface

* added tests

* fixed main setup

* Update how_to_define_looper_config.md

* Update __init__.py

* Update test_other.py

* Ise

* added changelog and minor naming changes

* remove old logging function

* dev version bump

* fix typo in html_report and upgraded pandas requirements for pephubclient

* fixed requirements

* fixed docs requirements

* added versioneer to doc requirements

* added Cython to doc requirements

* added readthedocs config

* added looper to requirements docs

* allow for using pipestat.summarize, align with pipestat 0.4.0

* clean up code, update usage doc

* update doc requirements pephubclient

* downgrade docs to 3.10

* adjust get_status to use proper sample_name if pipestat configured #326

* adjust conductor to retrieve pipestat manager variables with pipestat 0.4.0 refactoring.

* Allows skipping some tests if run offline. Closes #370

* work on using test_args instead of subprocesses

* Finish switching applicable tests away from subprocess

* Lint and update doc string to test_args_expansion

* Change return type.

* lint

* add test for var_templates #357, and clean up tests

* attempt simple check to see if provided pipelines are callable #195

* minor adjustments, polished docstring

* work on new peppy

* update changelog

* lint

* update version to 1.5.0

* update changelog

* update reqs and changelog to use pipestat v0.5.0

* more work on peppy without attmap

* Refactoring for looper config

* added looper config file argument

* code fix

* Added comment about deprecating for old looper specification

* fixed looper init error

* change logo for docs build tab icon

* fix favicon

* update docs and changelog for 1.5.0 release

* - fix `looper table` failing without sample.protocol , update change log for point release

* fix "--looper-config"

* update version and changelog date

* adjust pipestat requirement to be >=0.5.1

* lint

* clarify message on rereun

* dev version flag

* version bump to 1.5.2-dev

* fix indentation

* fix message logic

* improve rerun messaging

* use f strings

* default msg

* fix error message.

* clean up divvy docs. Fix #393

* add expanding paths in read_looper_config_file

* Is this the fix for #398?

* fix some item attr confusion

* lint

* Adjust dry run submission reporting

* oops

* more attr fixes

* lint

* initial poc for rewiriting classes to return dict for pytests

* more test changes

* test polish

* use logmuse for all logging msgs

* typo

* reduce runp and collator tests

* Skip Checker related tests until CheckerOld is deprecated

* remove divvy dependence on attmap

* expand submission paths correctly

* Update requirements pephubclient

* refactor CLI code

* clean up imports

* cli imports cleanup

* final cli polish, linting

* have self.debug use consts and clean up structure

* add imports to cli_divvy

* Add pipestat configuration exception to sub commands and tests

* first pass at refactoring prj.pipestat_configured_project prj.pipestat_configured

* fix utils import

* fix tests, lint

* add pipeline type for looper table

* add sample_level stats for looper table

* fixed relative path

* fixed tests and updatd documentation

* added docstring to test

* remove unused code

* tweak sample_level pipestat retrieval

* remove CheckerOld

* Fixed #410

* Fixed #395

* change destroy_summary to use pipestat configuration and pipestat functions

* add --project default for destroy command and rename funcs for disambiguation

* remove html_reports and update imports.

* add pipestat compatible pep for pytesting and associated test.

* adjust pytest fixture.

* add new pytests for pipestat configurations, re-implement Check tests

* change LOGGER.warn to LOGGER.warning due to deprecation

* change function name for copyfile, remove todos.

* attempt to change the way looper gets pipestat configuration, tests broken

* fix broken tests and file path issues

* fix config_file namespace and pep-config issue

* add plugin

* Pipestat polish (#412)

* fix pep-config issue

* pass output_dir to pipestat #411 and # 390

* adjust building looper namespace

* revert to using pipestat config path, tests broken

* fix tests by reverting some changes

* allow sample name to be passed during config check, raise value error if no pipestat config

* pass sample name only

* resolve schema path based on pipestat config

* clean test and allow pipestat namespace to include config_file path

* remove unnecessary pipestat namespace key value pairs

* Attempt constructing a pipestat config file from looper config and piface and then writing to file. Tests broken.

* fix tests

* general clean up

* remove sample name during pipestat creation

* remove redundancy

* lint

* clean up comments

* fix runp for pipestat and add to pytest

* add information to looper's pipestat documentation.

* Update changelog

* 406 relative path (#408)

* Changed relative path to project config

* lint

* fixed failing test

* fixed failing test

* merge from dev and fix conflicts

---------

Co-authored-by: nsheff <nsheff@users.noreply.github.com>
Co-authored-by: Donald Campbell <125581724+donaldcampbelljr@users.noreply.github.com>

* add info on looper report and point to pepatac example #41

* update looper to reflect pipestat's refactor from sample_name to record_identifier

* use pipestat to set status to 'waiting' if rerunning a failed sample. #169

* update pipestat and yacman versions to alpha and dev versions in requirements.

* update yacman to be released v0.9.2

* resolve pipestat module imports

* update eido 0.2.1 and peppy to pre-release version

* skip test failing on GitHub for now.

* modify test skip to include entire class.

* lint

* revert change per discussion: #420

* change naming of generated pipeline_interface.yaml #417

* add clarification in docs for accessing sample.sample_yaml_path and sample.sample_yaml_cwl #421

* move plugins to `plugins.py` #419

* Added writing output_schema and count_lines.sh when initializing a pipeline interface #418

* begin work on looper link #72

* continue looper link, now functional #72

* allow access for looper.pep_config #424

* fix accessing looper.pep_config #424, add building out looper namespace based on config file #423

* add better error message for #397

* fix pipestat import

* remove sample_name and project_name from pipestat_namespace in favor of record_identifier

* update pipestat req

* update pipestat req for newest alpha release

* update changelog.md

* update project definition docs

* update pipeline-interface-specification.md

* update parameterizing-pipelines.md and initialize.md

* misc documentation corrections

* update usage.md

* update hello-world example

* 2nd pass on docs

* add clarification for sample_modifiers and configuring project

* fix for #427

* implement path expansion during pipestat configuration check

* print link directory to terminal after using looper link

* print report directory to terminal after using looper report

* change schema_path to output_schema for pipestat config

* update docs schema_path to output_schema for pipestat config

* fix key error with populate_sample_paths

* allow rewriting looper config even if it exists

* only pass sample_name as record_identifier if it is given

* add default_project_record_identifier if using {record_identifier} in pipestat config

* fix #428

* fix pep_config not populating during runp

* remove redundant pep_config arg

* fix bug with path expansion during config read

* pass looper's samples to pipestat summarize

* WIP attempt at selector-flag #126

* more progress, works for selection #126

* Add tests, both selection and exclusion based on flags now works #126

* fix path issue for items with underscores #126

* ignore flags if slecting on flags, change sel and exc to have nargs="*"

* update docs

* add more tests for selecting attributes in tandem with flags #126

* fix project typo in docs

* version bump for prerelease 1.6.0a1 and add pipestat req v0.6.0a9

* 2nd attempt add pipestat req v0.6.0a9

* bump to 1.6.0a2

* fixes #430 and adds corresponding tests

* Updates for new peppy

* lint

* basic tab completion for initial commands #422

* polish docs for tab completion #422

* lint

* potential fix pepkit/peppy#459

* Revert "potential fix pepkit/peppy#459"

This reverts commit 281cef4.

* version 1.6.0a3 pre-release

* v1.6.0 release prep

---------

Co-authored-by: Khoroshevskyi <sasha99250@gmail.com>
Co-authored-by: Donald C <125581724+donaldcampbelljr@users.noreply.github.com>
Co-authored-by: ayobi <17304717+ayobi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@vreuter vreuter

Labels
likely-solved
Projects
None yet
Milestone
1.5.0
Development

No branches or pull requests
4 participants
@afrendeiro @nsheff @vreuter @donaldcampbelljr