Re-work on adding template support for Pinot Ingestion Job Spec #5372
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
mayankshriv
reviewed
May 12, 2020
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/IngestionJobLauncher.java
Outdated
Show resolved
Hide resolved
| public static Map<String, Object> getDefaultContextMap() { | ||
| Map<String, Object> defaultContextMap = new HashMap<>(); | ||
| Instant now = Instant.now(); | ||
| defaultContextMap.put("today", DATE_FORMAT.format(new Date(now.toEpochMilli()))); |
There was a problem hiding this comment.
Yes, we can add. What do you expect the value for LATEST? same as TODAY?
kishoreg
approved these changes
May 12, 2020
...-tools/src/main/java/org/apache/pinot/tools/admin/command/LaunchDataIngestionJobCommand.java
Show resolved
Hide resolved
mcvsubbu
reviewed
May 12, 2020
pinot-spi/src/main/java/org/apache/pinot/spi/ingestion/batch/IngestionJobLauncher.java
Show resolved
Hide resolved
xiangfu0
force-pushed
the
template_ingestion_spec
branch
from
dfbd372
to
077fe52
Compare
xiangfu0
force-pushed
the
template_ingestion_spec
branch
from
077fe52
to
aa54768
Compare
Codecov Report
@@ Coverage Diff @@
## master #5372 +/- ##
===========================================
- Coverage 66.44% 44.48% -21.97%
===========================================
Files 1075 1080 +5
Lines 54773 55123 +350
Branches 8168 8249 +81
===========================================
- Hits 36396 24519 -11877
- Misses 15700 28507 +12807
+ Partials 2677 2097 -580
Continue to review full report at Codecov.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the re-work of #5330, as the previous PR #5341 is reverted in #5366.
Templating is based on Groovy SimpleTemplateEngine.
we need to support Pinot Ingestion job take an ingestion job spec template file and passing thru values.
This will make user to schedule daily job simpler, without generating new yaml file everyday.
-propertyFileparameter inLaunchDataIngestionJobCommand(in format of 'k1=v1' 'k2=v2'...) which will be the context to set in the template.-valuesparameter inLaunchDataIngestionJobCommandwhich takes a list of parameters(in format of 'k1=v1' 'k2=v2'...) that will override the context read from thepropertyFileand set to the template.today,yesterdayinto context for dev simplicity.Usage example:
User can define
inputDirURIas'file:///path/to/input/${year}/${month}/${day}/${hour}'in job spec and a config file
job.configcontains:Then the new command line is :
After that the real ingestion spec passed to ingestion job will have
inputDirURIas'file:///path/to/input/2020/05/06/07'