Issue #8035 : Add Data set output transform - #8118
Merged
Merged
Conversation
Add a Data set output transform so pipelines can write rows into a unit-test data set (metadata + CSV) from Hop GUI and hop-run, without hard-coding the data set file path. The dialog is built from @GuiWidgetElement annotations with a grouped layout. Rows are streamed through DataSetCsvWriter, which is also used by Write rows to data set. Recreate builds the data set from incoming fields; validate checks an existing set before overwrite.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a Data set output transform so a pipeline can write rows into a unit-test data set (metadata + CSV) from Hop GUI and
hop-run, without pointing a Text file output at the data set path.This addresses #8035. The original request also asked for a “Current rows / All rows” choice on Create data set; that stays out of scope. Canvas Create data set and Write rows to data set already cover interactive capture of the full pipeline stream.
Transform
'${HOP_DATASETS_FOLDER}'when empty), and CSV filenameThe dialog is generated from
@GuiWidgetElementannotations (GuiCompositeWidgets.addScrolledComposite) with a grouped layout so the fields sit in a fill-and-scroll container above the button bar.Internals
DataSetCsvWriter(Apache VFS);DataSetCsvUtil.writeDataSetDataand Write rows to data set use it instead of buffering the whole streamDataSet:createFieldsFromRowMeta/validateRowMetaTests and docs
./mvnw -pl plugins/misc/testing test,apache-rat:checkon that module)main-0108-data-set-output(generate then read back against a golden set)GuiWidgetGroupType.BOXESis annotated on the dialog fields; it currently still renders as tabs until #8117.Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
mvn clean install apache-rat:checkto make sure basic checks pass. A more thorough check will be performed on your pull request automatically.git rebase -i.addresses #123), if applicable.To make clear that you license your contribution under the Apache License Version 2.0, January 2004
you have to acknowledge this by using the following check-box.