-
-
Notifications
You must be signed in to change notification settings - Fork 198
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Additional options for output file collection #134
Milestone
Comments
@StarvingMarvin how about this? We add a special case outputBinding:
glob: **
outputEval: |
${
var ret = [];
for (var i = 0; i < self.length; i++) {
// only capture .txt files
if (self[i].path.match(/\.txt$/) { ret.push(self[i]); }
}
return ret;
} |
+1 |
We should get the major changes in Draft 3 finished first; I am moving this to the Draft 4 milestone. |
According to @tetron this will be obsoleted by DirectoryType |
tetron
pushed a commit
that referenced
this issue
Mar 21, 2018
27c20e8 Corrects initializer to FileCache when no TMP or HOME (#159) 66457f2 unlike cwltest, only one wheel here (#162) a343bc8 main: print string instead of bytes (#146) d42ac08 Travis fix for forks (#161) f74a314 Fix date --utc. Update version for next release. Update classifiers. (#160) 6ce2b9a Default field (#150) 8e520ec correct path due to schema_salad vs. schema-salad (#154) c359d07 Upgrade release process (#153) 5d75f90 Validation shouldn't warn about foreign properties if skip_schemas is True. (#152) 9b234bb Make link checking $schemas section and extension fields non-fatal. (#145) 8752f87 Make exceptions loading extension schemas non-fatal. (#144) 8ae7664 Unify the error message format for invalid YAML (#143) fc00a2c Add space between filename and message for `--print-oneline` (#142) 91e1b41 Fix error message for resolve_ref with --print-oneline (#141) 3f829b1 Better itemize handling for --print-oneline (#136) 7b1d94f include schema_salad/tests/docimp in the dist (#140) 2d72415 Fix for pylint (#139) 6e9abce Fix for invalid YAML with `--print-oneline` (#137) a9db248 Merge pull request #138 from tom-tan/add-test-for-print-oneline 25fd531 Add test for Issue #135 c29eac3 Python schema loader codegen (WIP) (#134) d654f83 Merge pull request #135 from tom-tan/print-oneline 52fd416 Fix for pep8 d6e7298 Fix for pylint cded5f4 Add --print-oneline option c7f3140 Refactor Windows relative path support (#131) 2d94392 Add include_traceback option to SourceLine. (#133) 70519d9 Merge pull request #128 from common-workflow-language/bump_avro 2543f2c Bump version of `avro` for Python 3 9554e5f Merge pull request #127 from common-workflow-language/py3-win-workaround f637d60 add note about workaround eac115d Merge pull request #125 from common-workflow-language/tmp_fix_windows 25990c5 __init__.py: Use logger to raise error on Windows rather than print 36c04d1 utils.py: Fix missing import ce49d5d add error message when not able to create \tmp folder in Windows a81a35f schema_salad/__init__.py: Create '\tmp' folder on windows if not present f61676e utils.py: add onWindows() function 5e49770 Merge pull request #123 from common-workflow-language/modernize 1df0a01 Merge branch 'master' into modernize f1aea74 requirements.txt: add future module dep 5cd2f36 ignore type checking past module 3ad9a21 remove --warn-unused-ignores flag since past is giving error in py3 but not py2 a6cedd3 move avro typshed to 2and3 and use Text instead of unicode 6450eee delete typesheds/3/avro-python3 193ad1e fix avro requirements related issue 627f1d4 fix linting errors 7bf4b32 setup.py: for python3 use forked avro pip package b5b9595 setup.py: dependency link hack, but need to pass argv to pip explicitly 043bb3a schema.py: remove redundant lines while making avro lib import 626208d __init__.py: call autotranslate on avro py2 if running in python3 f619227 Merge pull request #122 from common-workflow-language/modernize de7719e use io.open instead of builtin open Leads to more consistant behaviour across py2 and py3 also, specify encoding in the open command rather than reading raw bytes and manually decoding 58f1daa Merge pull request #121 from common-workflow-language/modernize 540f0fd fix: use 'rb' instead of 'wb' since we're reading the file a46401a Merge remote-tracking branch 'origin/master' into modernize ecf4c3f ref_resolver, makedoc: explicitly open files in binary mode d47200e mypy_requirements: bump mypy version to 0.520 eef27fa Merge pull request #120 from kapilkd13/windows-fix eb3f5e2 Merge branch 'master' into windows-fix 4b4c4d6 modified urljoin: on windows it was ignoring Drive names when constructing final url 3cba9dd Merge pull request #110 from kapilkd13/windows c51b0e1 re-add six import f91d588 Merge branch 'master' into windows 517617c Merge pull request #119 from common-workflow-language/tweak-build dbe4c81 tweak the build 7ea7dcf Merge pull request #118 from common-workflow-language/manu-chroma-patch-1 6c19085 py3 compatibility: modified url2pathname acoordingly b54bd29 py3 compatibility: modified urlparse acoordingly bd75357 modifying urljoin for same url and baseurl, moving windows path conversion to after all resolution point 7b614ea modified test mixin for correct base path 5188caa added windows build badge 1be2b8f added unix build badge 45b1372 windows compatibility issue: windows file paths were not fetched b54ceea windows compatibility issue: windows Drive names are considered as scheme 3834118 windows compatibility issue: after path split windows path contain / before drive name 89c48b2 test_examples: normalizing path and correcting file scheme for windows 46afdf7 normalizing path depending on OS f047c04 test_fetch: correcting file scheme for windows 837e3e1 added appveyor.yml for windows testing and Integration bde8dfb pin upper version of ruamel.yaml package 4d1c3ff Merge pull request #117 from common-workflow-language/better-bump a8c3fde oops b3c579f a better bump of the version (mea culpa) fa93acc Merge pull request #109 from manu-chroma/modernize 52c054e 3.6 too! a8903de setup.py: change version, uncomment classifiers for py3 d9bdc8b setup.py: cleanup 7748526 schema.py: fix some types, remove redundant str b2a5a26 bump ruamel version, put upper bound on ruamel in setup.py 3d60033 mypy_requirements.txt: bump mypy version a1f697b ref_resolver.py: mypy: use Text instead of unicode 669ebfc remove redundant condition with typing package 6b3d068 typeshed/3: add stubs(autogen) for avro-python3 8672a5c sourceline.py: replace AnyStr with Text type 740e235 mypy: turn off warn for unused ignore in --py3 4f0ad4f tox: separate out mypy targets for py2 and py3 d16c0e5 move cachecontrol stub files to typshed/2and3/ ad05e61 minor changes in rdflib stub files to make them compatible with mypy in py3 mode and move to typshed/2and3/ directory 65e583c mypy: some improvements in type annotations f051f49 typesheds/2.7: remove stub files which already come with mypy==0.511 1d11190 Makefile: add target for mypy without --py2 flag c6c88f1 add mypy.ini and ignore all ruamel.yaml package errors 2850043 makedoc.py: pass unicode string mintune markdown function 963ea3f mypy: add typeshed/2and3 in mypypath 62cbece add stub file for mistune package author = "Aleksandr Slepchenkov" email = "Sl.aleksandr28@gmail.com" 4958ebf use Text type where ever Union[str, unicode] is expected d16a308 mypy: remove redundant ignore type annotations 090c1ba mypy: use latest stable version of mypy - use update ruamel.yaml package which includes more mypy annotations 67d7481 use same import pattern for six.urllib accross the codebase - also, re-ogranise six lib imports 150730c py3 fix: use six StringIO import 0773e44 fix urlparse using six in tests and schema_salad core; no automated fixers 6a11b16 mypy type annotations fix: replace unicode -> Text 8b6f9b3 Apply Python3 modernize transforms 1eb19b6 use typing>=3.5.3 b0b1f14 disable git version tagging to avoid testing conflicts with cwltool ff000db setup.py: add six as explicit dependency c66a3f9 tests: remove redundant parentheses in print statements b1605e3 fix: use six integer_types and string_types in place of int and basestring 29dcb7f tox.ini: fix bugs, enable flake8 linting for all python versions 5efed90 tox.ini: re-write file, include unit tests for all supported versions of python - enable more python3 versions on travis for unit testing 508c8d3 setup.py: bump version to '3.0' 9874d11 use same import pattern for six.urllib accross the codebase - also, re-ogranise six lib imports d23c804 tests: use assertEqual() inplace assertEquals() 3095597 fix: use six.integer_types to handle int, long incompatibles 965378f .tox.ini: turn on python3 unit testing 7ee4848 .gitignore: add more local folders 4d3c6de fix: use list of dict keys to iterate, fix flake8 warnings - py3 compatibility 916aaa1 py3: use six.moves.urllib instead of urllib 9108c7b fix: minor: py3 compatibility wrt avro cec0213 setup.py: install different version of avro in case of py3 runtime 3704c98 py3 fix: use six StringIO import e795ac9 fix: make regex command python3 compatible b4273f2 fix urlparse using six in tests and schema_salad core; no automated fixers df685ea modernize tests e9bb3ec mypy type annotations fix: replace unicode -> Text 3080c69 Apply Python3 modernize transforms caec629 Merge pull request #116 from manu-chroma/master 5a76bb6 create mypy_requirements.txt for mypy related deps ba88f5e add mypy, typed-ast dependency in requirements.txt b61311c Makefile: pin mypy, typed-ast version for jenkins make target 9d9317e Merge pull request #114 from manu-chroma/master 8372fe2 Merge branch 'master' into master d370602 Merge pull request #115 from common-workflow-language/skip_schemas e39018a Optional skip_schemas 6a582c3 refactor flatten function. move to utils.py 3bba34e minor refactor: create utils.py 5b3863a Merge pull request #112 from common-workflow-language/tweak-cov ffc3367 exclude tests from coverage report 5b0b019 Validation fix. Must raise exception when raise_ex is true and record class (#107) 424ad78 Fix for self-colliding ids on $import (#102) 8a6eaff Validating Ids for duplicate Issue#56 (#98) a560ef3 Pass through logger to capture warnings. (#101) ca9218e Merge pull request #99 from manu-chroma/master a63f75f schema_salad/tests: add tests for cli args testing: --version, empty args - fix return condition in main() 04c6784 schema_salad/main.py: --version now correctly prints version 61c2203 Merge pull request #100 from manu-chroma/test_patch a082ab5 schema_salad/tests: use print() in tests 0114153 In vocab flag (#97) 39516e5 Add map and typedsl resolution documentation. (#96) 0b76655 Merge pull request #92 from kapilkd13/master c00ad4c Merge branch 'master' into master 2114715 Merge pull request #93 from common-workflow-language/mypy-0.501 f1d824d Don't switch to mypy v0.501 yet 8ab39f9 Upgrade to mypy v0.501 --strict 4f4fe73 added instalation from source instructions d2d7b1e added a warning message when error is being ignored 1e05b44 Set publicID when passing plain data to RDF parser. (#91) 15c0ab8 Relax strict checking foreign properties (#90) 3a7cf29 Merge pull request #89 from tjelvar-olsson/master f16c1a8 Add more directory fallbacks Loader session 50d0361 Make ref_resolver.Loader more fault tolerant 7522a24 Make split_frag optional in converting fileuri. (#88) 878a74f Merge pull request #87 from common-workflow-language/release-update 6209496 newer pip dependency 0347b8e Merge pull request #85 from chapmanb/debug-fixes 5c4ef60 Merge branch 'master' into debug-fixes 286001f Merge pull request #86 from common-workflow-language/mypy-470 9676ea8 track mypy rename e47f4e6 Improve debugging: missing filename and bad lists bee8bf1 Merge pull request #84 from common-workflow-language/mypy0.470 822c8c1 upgrade to mypy 0.470 42f4bdd Fix performance regression in schema loading by stripping ruamel.yaml metadata, (#83) d21fa84 Expand uris in $schemas in resolve_all(). (#82) 55018dd Bump version to 2.2 (#81) b883526 Bugfix to uri_file_path. (#80) 0c7cc02 Remove pathlib2 because it is only used for one thing but creates dependency issues. (#79) f31369c Merge pull request #78 from common-workflow-language/explict-include f34d4ac explicitly include MD files & fix release test c1cf3e9 Merge pull request #77 from common-workflow-language/simplify-manifest a2e4236 simply the manifest 5d04ddb Merge pull request #76 from common-workflow-language/ensure-test-paths a37b623 ensure all test data is loaded flexibly 6a02920 Merge pull request #75 from common-workflow-language/adjust-manifest cefb22e Merge branch 'master' into adjust-manifest dd496a1 Include all relevant files so tests work post installation 8edfbc2 Bugfix resolving references to document fragments. (#74) a3ba891 Relax ruamel.yaml version pin. (#73) a5bbb36 Ensure keys are sorted when generated CommentedMap from a regular dict. (#72) ebdf27d Merge pull request #71 from common-workflow-language/pip-conflict-checker 2aecb05 Merge branch 'master' into pip-conflict-checker 999d689 Merge pull request #70 from psafont/win32 01af23d Fix typing 7a793ac drop html5lib pin, add pip-conflict-check 0e616d1 Fix types ad742f3 Manage pathlib and pathlib2 for all python versions cb754df Make url2pathname args compatible with python3 da751f7 Don't assume the type of ref f302f5f Fix a couple of test cases 0c0e4cc Added pathlib2 requirement 159541c Paths and URIs conversions are more generalized 9a9d639 Tweak cmap to better propagate filename down. (#69) 5a15c58 merge 1.21 changes into salad 2.0 (#68) a1db0ac Merge pull request #67 from common-workflow-language/mr-c-patch-1 56ac12b Link to v1.0 specs beed46d Merge pull request #65 from common-workflow-language/mobile-friendly 1e0f6a5 set the meta viewport tag f0b88d0 Bump major version number, because of backwards-incompatible changes. (#63) 8ac4540 Provide line numbers in error messages (#62) 8454546 Bugfix for validating "class" field. (#61) 01dd303 Optimize validate (#60) 423e48b Merge pull request #58 from common-workflow-language/quiet-external-schema f2d86c1 Merge branch 'master' into quiet-external-schema f6744ad Merge pull request #53 from common-workflow-language/check-file-support-http 1728fd4 fix types 823f378 Merge remote-tracking branch 'origin/master' into check-file-support-http 9b6a6be demote log about external schema b1d7b69 Support http/https existence check (using HTTP HEAD) in check_file. git-subtree-dir: v1.1.0-dev1/salad git-subtree-split: 27c20e89a8df4ff8cdb28fae59d2a5ea652c0afc
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Right now the only way to specify which files goes to which output are globs, but we had issues with tools where ordinary globs weren't enough to distinguish between different outputs (e.g filenames without "something"). We kinda solved our problem with bash extended globs, but it's not the cleanest solution (we have to run it inside a docker container), and it is not really expected that other implementations will handle them. Adding another option in CommandOutputBinding like
"regexp": "<something>"
would probably suffice. Finally we could go "full Turing-complete" and add an option like"filter": {expression}
that could get list of all files from working dir and return whatever files appropriate for that output.The text was updated successfully, but these errors were encountered: