DF/data4es: Log diversification #375
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
anastasiakaida
force-pushed
the
data4es-log-differentiation
branch
from
6489bfd
to
a73e169
Compare
STDERR wrapper is aimed to catch all messages from STDERR,
add to each one -- line by line -- the name of log producer
(script name) and redirect messaged to STDERR again.
It is considered to apply the following function for processing
chains the following way:
err_wrepper param1 param2 ... param*
Where a set of parameters can be presented as a sequence
kept in one variable like this:
"${base_dir}/../093_datasetsFormat/datasets_format.py -m s"
It doesn't matter how many variables will be passed -- it is supposed that
script name will be taken from the first argument.
The function is applied to all commands marked as "$cmd_%stage_number%" in following chains: *Sink chain *Subchain 17 *Subchain 91 *Source chain
anastasiakaida
force-pushed
the
data4es-log-differentiation
branch
from
7333aed
to
1504150
Compare
anastasiakaida
changed the title
anastasiakaida
changed the title
We need to return not only a variable value, but its name also.
The wrapper function now adds info about a stage, not only just bare name of any script.
|
Couldn't make it work properly (see below). Did I miss something? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current situation about how to define the log's producer (stage's script) is unclear.
To solve this issue we propose a solution that allows keeping stages scripts as they are.
The supervising script (data4es-run) runs processing chains so even if any sequence is changed (some stages are removed, changed, added) it would be easy to keep the way of logging the same.