Fixed bugs in recursion depth and added a path example #761
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of what I changed
This fixes some of the issues mentioned in #757. It also adds a path example where the recursion depth limit is disabled. The example is commented out until we decide we really want to support this pattern. If we do, then those exceptional paths should be exposed as configuration parameters.
E2E test
TESTED:
Ran the pipeline with a sample QuestionnaireResponse that had deep
item.item.item...
recursions and confirmed that all of theanswer
fields are transformed into the Parquet file without increasing the general recursion max-depth, if that path is provided as an exception.Also tested the same scenario by increasing max-depth and no exceptional paths; at levels 1, 2, 3, etc. the expected fields were present.
Checklist: I completed these to help reviewers :)
I have read and will follow the review process.
I am familiar with Google Style Guides for the language I have coded in.
No? Please take some time and review Java and Python style guides.
My IDE is configured to follow the Google code styles.
No? Unsure? -> configure your IDE.
I have added tests to cover my changes. (If you refactored existing code that was well tested you do not have to add tests)
I ran
mvn clean package
right before creating this pull request and added all formatting changes to my commit.All new and existing tests passed.
My pull request is based on the latest changes of the master branch.
No? Unsure? -> execute command
git pull --rebase upstream master