New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable sending multiple file targets based on a glob pattern #30

Open
samuell opened this Issue Jun 9, 2016 · 3 comments

Comments

Projects
None yet
1 participant
@samuell
Member

samuell commented Jun 9, 2016

This is needed for split operations. E.g:

split := scipipe.NewFromShell("split", "split -d {i:infile} {o:slitfiles}")
split.GlobOutputs("splitfiles", "output.txt_split*")
@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jun 9, 2016

Member

Maybe rather:

split := scipipe.NewFromShell("split", "split -d {i:infile} {g:splitfiles}") // Use new 'g' operator for glob patterns
split.SetPathStatic("splitfiles", "output.txt_split") // Use the normal path setters to define the path for the prefix

or

split := scipipe.NewFromShell("split", "split -d {i:infile} {glob:splitfiles}")
split.SetPathStatic("splitfiles", "output.txt_split") // Use the normal path setters to define the path for the prefix
Member

samuell commented Jun 9, 2016

Maybe rather:

split := scipipe.NewFromShell("split", "split -d {i:infile} {g:splitfiles}") // Use new 'g' operator for glob patterns
split.SetPathStatic("splitfiles", "output.txt_split") // Use the normal path setters to define the path for the prefix

or

split := scipipe.NewFromShell("split", "split -d {i:infile} {glob:splitfiles}")
split.SetPathStatic("splitfiles", "output.txt_split") // Use the normal path setters to define the path for the prefix
@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jun 27, 2016

Member

Some considerations I realized today:

  • The main problem is that you can't, or will not typically, specify the output
    files explicitly, but rather will provide a base name (optimally), or some
    kind of pattern, which the outputs will be based upon. Sometimes you don't
    even do that. Thus, the globbing pattern has to be stated explicitly.
    • One needs to create an outport though. It has to be after an # in the
      command though.
    • One should be able to provide a globbing pattern directly inline in the
      command, similar to how we thought to do this with normal output patterns.
      So, for example, we could have: split {i:in} basename # {g:splits:basename*}
  • Just realized that you'll have to use an anonymous function to generate the
    glob patterns too, since they typically depend on input patterns.
  • It is probably better though to go one step further, though, and make a
    general function for producing outputs, so that you'll have to call glob and
    produce filetarget yourself.
    • If allowing the short-form, in-line, definition as shown above, you could
      avoid needing low-level coding for most tasks.
Member

samuell commented Jun 27, 2016

Some considerations I realized today:

  • The main problem is that you can't, or will not typically, specify the output
    files explicitly, but rather will provide a base name (optimally), or some
    kind of pattern, which the outputs will be based upon. Sometimes you don't
    even do that. Thus, the globbing pattern has to be stated explicitly.
    • One needs to create an outport though. It has to be after an # in the
      command though.
    • One should be able to provide a globbing pattern directly inline in the
      command, similar to how we thought to do this with normal output patterns.
      So, for example, we could have: split {i:in} basename # {g:splits:basename*}
  • Just realized that you'll have to use an anonymous function to generate the
    glob patterns too, since they typically depend on input patterns.
  • It is probably better though to go one step further, though, and make a
    general function for producing outputs, so that you'll have to call glob and
    produce filetarget yourself.
    • If allowing the short-form, in-line, definition as shown above, you could
      avoid needing low-level coding for most tasks.
@samuell

This comment has been minimized.

Show comment
Hide comment
@samuell

samuell Jun 18, 2018

Member

Now with #41 implemented, it should be easy to support setting output patterns, like:

split := wf.NewProc("split", "split {i:in} some_prefix_")
split.SetOut("split_files", "some_prefix_*")
Member

samuell commented Jun 18, 2018

Now with #41 implemented, it should be easy to support setting output patterns, like:

split := wf.NewProc("split", "split {i:in} some_prefix_")
split.SetOut("split_files", "some_prefix_*")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment