Generic source and sinks?

[drghan]

I've been using ETL/ELT tools for many years, and started trying to build some things with Duckle. It looks very promising!

I'm not sure if it is already possible to make the source and sink locations more dynamic. Let's say I want to add a timestamp to a sink path, how would I do that?

Also, let's say I have a database source containing table names. I would the like to read the table names from this source, and use this to feed into a source component.
For example: a source containing a table with three rows: "tableA", "tableB", "tableC". If I read these three rows, in my pipeline I want to loop over these rows, and then use the values in these rows as the source table name in my source component. Is that possible?

Another small thing: if I have a pipeline in my project, can I move it to a folder in the project? It seems I'm not able to drag and drop.

[SouravRoy-ETL]

Thanks for the kind words, and these are three good questions. All of it is now supported on main and ships in the next build. Here is how each one works.

1. Dynamic source / sink paths (timestamps)

There are now run-time placeholders you can drop into any path field (source or sink). They are resolved fresh on every run, in UTC:

Placeholder	Example value	Notes
${date}	2026-06-17	calendar day
${time}	143052	HHMMSS
${datetime}	2026-06-17_143052	filename-safe, no colons
${timestamp}	1781794252	epoch seconds
${now}	2026-06-17T14:30:52Z	ISO-8601 (has colons, so better for a value than a filename)

So a sink path like:

${workspace}/exports/${date}/orders_${datetime}.parquet

writes to .../exports/2026-06-17/orders_2026-06-17_143052.parquet. The dated sub-folder is created automatically if it does not exist yet. ${workspace} (the project root) is also available.

These resolve the same way whether you run from the canvas, from a schedule, or headless with the runner / a built bundle, and a built bundle stamps the time when it RUNS (not when it was built), so a daily run lands in a new dated folder each day. When you type a placeholder into a field the panel shows you what it currently resolves to, so you can sanity-check it.

2. Data-driven source table names (loop over rows)

This works today with the Foreach control component (ctl.foreach):

1. Add the source that returns your table names (a table/query with one row per name, e.g. a column table_name holding tableA, tableB, tableC).
2. Wire that into a Foreach node and point its child-pipeline picker at a second pipeline that does the actual load.
3. In that child pipeline, set the source component's table-name property to ${ITER_ITEM_TABLE_NAME}.

Foreach reads the upstream rows and runs the child pipeline once per row, exposing every column of the current row as ${ITER_ITEM_<COLUMN>} (the column name upper-cased), plus ${ITER_INDEX} for the row number. So column table_name becomes ${ITER_ITEM_TABLE_NAME}. You can set the child to run rows sequentially or with a concurrency, and you can combine this with the placeholders above to write each table to its own dated file.

3. Drag a pipeline into a folder

This was a gap, and it is now added. In the Project tree you can drag a pipeline (or any item, or one of your own folders) and drop it onto a folder or the project root to move it. The target folder highlights while you drag over it, the move is rejected if it would put a folder inside itself, and it is saved to the project automatically.

If any of these does not behave the way you expect once you are on the new build, please let us know.

[drghan]

Sounds great, thanks a lot!
I'll try it as soon as the new build appears.

[SouravRoy-ETL]

Check it out! It is released.