New outputKey() and keepInputData() step methods #64
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keepInputData() can be called on any step, to add data from the input of a step, to the resulting outputs. It combines the data in an array. If the input was non array, you can define a key for that input. Same for non array outputs.
Group steps can now only produce combined outputs, as previously done when `combineToSingleOutput()` method was called. The method is removed. This was done because I don't really see any actual use case for a group with all the steps yielding separate outputs. Using the new `outputKey()` method, a step will convert non array output to an array and use the key provided as argument to this method as array key for the output value.
otsch
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New methods
outputKey()andkeepInputData()that can be used with any step. Using theoutputKey()method, the step will convert non array output to an array and use the key provided as an argument to this method as array key for the output value. ThekeepInputData()method allows you to forward data from the step's input to the output. If the input is non array you can define a key using the method's argument. This is useful e.g. if you're having data in the initial inputs that you also want to add to the final crawling results.Breaking change: Group steps can now only produce combined outputs, as previously done when
combineToSingleOutput()method was called. The method is removed.This is done because I think there is no real use case for a group step yielding separate outputs for all the different steps it contains. And on the other hand there have already been some Exceptions that have been thrown when certain methods are called without calling
combineToSingleOutput()first, because they can only work with combined output. So, this also removes the need for such Exceptions.