Skip to content

Issue #8035 : Add Data set output transform - #8118

Merged
hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-8035
Aug 27, 2026
Merged

Issue #8035 : Add Data set output transform#8118
hansva merged 1 commit into
apache:mainfrom
mattcasters:issue-8035

Conversation

@mattcasters

Copy link
Copy Markdown
Contributor

Adds a Data set output transform so a pipeline can write rows into a unit-test data set (metadata + CSV) from Hop GUI and hop-run, without pointing a Text file output at the data set path.

This addresses #8035. The original request also asked for a “Current rows / All rows” choice on Create data set; that stays out of scope. Canvas Create data set and Write rows to data set already cover interactive capture of the full pipeline stream.

Transform

  • Data set name, optional folder ('${HOP_DATASETS_FOLDER}' when empty), and CSV filename
  • (Re)create data set from incoming row metadata (default on)
  • Validate incoming field count, names (same order, case-insensitive), and types against an existing data set
  • Pass-through output; CSV is overwritten each run; multiple copies are rejected
  • Empty streams still write a header-only CSV

The dialog is generated from @GuiWidgetElement annotations (GuiCompositeWidgets.addScrolledComposite) with a grouped layout so the fields sit in a fill-and-scroll container above the button bar.

Internals

  • New streaming DataSetCsvWriter (Apache VFS); DataSetCsvUtil.writeDataSetData and Write rows to data set use it instead of buffering the whole stream
  • Schema helpers on DataSet: createFieldsFromRowMeta / validateRowMeta

Tests and docs

  • Unit tests for serialization, schema validation, and CSV round-trip (./mvnw -pl plugins/misc/testing test, apache-rat:check on that module)
  • Integration test main-0108-data-set-output (generate then read back against a golden set)
  • User-manual page plus nav / data-set / unit-testing docs

GuiWidgetGroupType.BOXES is annotated on the dialog fields; it currently still renders as tabs until #8117.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Run mvn clean install apache-rat:check to make sure basic checks pass. A more thorough check will be performed on your pull request automatically.
  • If you have a group of commits related to the same change, please squash your commits into one and force push your branch using git rebase -i.
  • Mention the appropriate issue in your description (for example: addresses #123), if applicable.

To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.

Add a Data set output transform so pipelines can write rows into a
unit-test data set (metadata + CSV) from Hop GUI and hop-run, without
hard-coding the data set file path.

The dialog is built from @GuiWidgetElement annotations with a grouped
layout. Rows are streamed through DataSetCsvWriter, which is also used
by Write rows to data set. Recreate builds the data set from incoming
fields; validate checks an existing set before overwrite.
@hansva
hansva merged commit 286b0f6 into apache:main Aug 27, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants