DF: allow single dataflow writing to two different ES indices. #366
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
For now it is pretty much hardcoded: stage 019 (esFormat), that
prepares data for upload, is only aware of 2 indices: "tasks" (for tasks
metadata) and "progress" (for tasks processing progress metadata).
If no index specified in the input data, "tasks" index is used.
There's also a "default" index -- it will be used if a desired index is
not configured in the provided (or default) config file.
It may be improved later, e.g. when PHP script starts reading the
configuration INI file by itself -- but for now that's enough.
ES configuration file previously contained only one index name
("ES_INDEX" variable); it is used by multiple stages so it is left in
the config file to keep existing functionality. It can not be set to the
"value of ES_INDEX_TASKS" like this:
ES_INDEX=$ES_INDEX_TASKS
Although it will be OK for shell scripts that use this file by simply
sourcing it, non-shell ones have to "manually" parse it, and
"$ES_INDEX_TASKS" will be considered as a plain string, not variable
value.
4 tasks
2 tasks
anastasiakaida
approved these changes
Jun 24, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For now it is pretty much hardcoded: stage 019 (esFormat), that prepares data for upload, is only aware of 2 indices: "tasks" (for tasks metadata) and "progress" (for tasks processing progress metadata). If no index specified in the input data, "tasks" index is used.
There's also a "default" index -- it will be used if a desired index is not configured in the provided (or default) config file.
It may be improved later, e.g. when PHP script starts reading the configuration INI file by itself -- but for now that's enough.
ES configuration file previously contained only one index name ("ES_INDEX" variable); it is used by multiple stages so it is left in the config file to keep existing functionality. It can not be set to the
"value of ES_INDEX_TASKS" like this:
ES_INDEX=$ES_INDEX_TASKS
Although it will be OK for shell scripts that use this file by simply sourcing it, non-shell ones have to "manually" parse it, and "$ES_INDEX_TASKS" will be considered as a plain string, not variable
value.
NOTE: after this PR all stages but Stage 019 will continue to use the
ES_INDEXparameter, while Stage 019 will ignore it.