Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update call> operator to read workflow files in a subdirectory #254

Closed
wants to merge 1 commit into from

Conversation

frsyuki
Copy link
Member

@frsyuki frsyuki commented Sep 1, 2016

Dividing a large workflow definition into multiple files is helpful to
organize a complex workflow definition. call> is more straightforward
(and less hacky) compared to !include in terms of syntax.

This changes call> operator to be able to load workflow definition from a file so that it can call workflows in a subdirectory. *.dig files in subdirectories won't be recognized as workflows.

This change includes removal of params argument from
ConfigLoaderManager.loadParameterizedFile method. params argument was necessary when it was using jinja2 to preprocess a file. It's unnecessary any more since digdag evaluates parameters when a workflow runs.

Dividing a large workflow definition into multiple files is helpful to
organize a complex workflow definition. `call>` is more straightforward
(and less hacky) compared to `!include` in terms of syntax.
@frsyuki
Copy link
Member Author

frsyuki commented Sep 1, 2016

A discussion is whether _workdir should be set or not.

For example, with following files, root.dig uses call> to call sub.dig, and sub.dig uses query.sql and nested.dig files:

root.dig
sub/sub.dig
sub/queries/query.sql
sub/nested/nested.dig

If call> operator doesn't set _workdir (current implementation), sub.dig will use relative path names like this:

td>: queries/query.sql
call>: nested/nested.sql

If call> operator sets _workdir to sub/, sub.dig will use absolute path names like this:

td>: sub/queries/query.sql
call>: sub/nested/nested.sql

@frsyuki
Copy link
Member Author

frsyuki commented Sep 1, 2016

Setting _workdir lets users run a subworkflow individually. hm...

@danielnorberg
Copy link
Contributor

I believe a workflow file should refer to files relative to the location of the workflow file.

I.e. in the above example, nested.dig should refer to query.sql using the path ../queries/query.sql.

When resolving the file path, resolve it in the directory of the workflow file. This way nested workflow files can be called by other workflows in other directories using call> as well as run directly by digdag run.

digdag run itself should look for the .digdag directory in cwd (or a parent directory) according to #70 and use the directory containing .digdag as the project root. File references should not be allowed to go outside the project directory.

@frsyuki
Copy link
Member Author

frsyuki commented Sep 16, 2016

Updated implementation is #290.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants