Handing many output files from a single input file

[namurphy]

I read in the (wonderful and greatly appreciated!) documentation that:

A processing code must have a command-line interface and produce a single output file from one or more inputs.

Suppose there's one step in a data pipeline where the input is one file and the outputs are many files. Can dbprocessing handle situations like this? Or alternatively, are there workarounds?

I'll be talking with @jtniehof in a few days (yay!) so we can chat in more detail then about how to handle this inverse e pluribus unum step. Many thanks!

[jtniehof]

Processes are definitely set up as many products to one product. Obviously one-to-one is a special case of this (as are the none-to-one and many-to-none variants of the DBRunner generation and RUN timebase, both added later). In the original design decision, we went over this pretty carefully and decided that this limitation was worth it.

There isn't a good workaround as such. There are several "maybe you can think about the problem differently" options:

1. True workaround: have the code create a bunch of files and throw them in incoming directly, but then you lose provenance.
2. Set up the code such that, even if there's a bunch of shared processing, there's only a single output each time it's called. Then set each up as a separate output product for each and accept some inefficiency from running the code several times. You can have a single actual piece of code (e.g. Python package) which is treated as different Codes in the database based on flags. That information often removes one of the perceived needs for this setup.
3. Are they really different outputs, or should they be combined into a single file? I've seen cases where people are (IMO) way too granular in their CDF files (apply freely to other self-describing formats), e.g. putting each species in a separate file.
4. Set up an intermediate product that contains the shared results.

I'm not opposed to doing this. We'd have to make quite a few changes...the output information on a process would be via a new linking table instead of a single column, and there's a fair bit of dependency calculation work to be managed.

There's also the question of multiple output files of a single product, e.g. if taking a month of input and making daily outputs. That's probably even harder.