GOBBLIN-1462: Ensure FsSpecConsumer handles config keys which are prefixes of other keys#3302
GOBBLIN-1462: Ensure FsSpecConsumer handles config keys which are prefixes of other keys#3302sv2000 wants to merge 2 commits intoapache:masterfrom
Conversation
| jobSpecBuilder.withJobCatalogURI(avroJobSpec.getUri()) | ||
| .withVersion(avroJobSpec.getVersion()) | ||
| .withDescription(avroJobSpec.getDescription()) | ||
| .withConfigAsProperties(props) |
There was a problem hiding this comment.
Do we still need this line?
There was a problem hiding this comment.
Yes, we will still need this line. The only change is we are now explicit about how the config member variable of JobSpec is set. Previously, it defaults to using ConfigUtils#propertiesToTypedConfig() which throws an exception when it encounters prefixed keys.
There was a problem hiding this comment.
This API is quite confusing ... But I get the point why both will be needed.
Codecov Report
@@ Coverage Diff @@
## master #3302 +/- ##
============================================
- Coverage 46.46% 43.00% -3.47%
+ Complexity 10023 9316 -707
============================================
Files 2037 2041 +4
Lines 79291 79352 +61
Branches 8840 8845 +5
============================================
- Hits 36845 34124 -2721
- Misses 39031 41990 +2959
+ Partials 3415 3238 -177
Continue to review full report at Codecov.
|
| jobSpecBuilder.withJobCatalogURI(avroJobSpec.getUri()) | ||
| .withVersion(avroJobSpec.getVersion()) | ||
| .withDescription(avroJobSpec.getDescription()) | ||
| .withConfigAsProperties(props) |
There was a problem hiding this comment.
This API is quite confusing ... But I get the point why both will be needed.
sv2000
force-pushed
the
typesafeConfigPrefix
branch
from
ca2940b to
2e77de0
Compare
4 participants
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!
JIRA
Description
Currently, the FsSpecConsumer fails when building a JobSpec if the configuration contains keys which are prefixes of other keys, which is not allowed in typesafe config. In practice, this constraint may be violated particularly when loading system properties which are outside application's control. To ensure these properties are safely handled, the FsSpecConsumer should use the proertiesToConfig() method in PropertiesUtils which adds a suffix to the root properties.
Tests
Modified unit tests
Commits