-
Notifications
You must be signed in to change notification settings - Fork 621
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Manifest data in weblog message #1076
Comments
@johandahlberg @apeltzer @ewels any suggestions to this? The current PR #1077 only sends the manifest data on workflow submission, which I think is sufficient, as you can store the manifest data with the uuid4 of the run and link further trace messages. What do you think? |
Sounds great! Yes I think that’s probably sufficient. If we’re sending this, why not just send the entire config output on the workflow start? |
Sounds reasonable, though I don't know enough about the inner workings of nextflow to know the details of how this could best be achieved. 😄 I agree with @ewels that it might be useful to get the full config submitted at the start of the workflow. |
is there's a really need for that? I would prefer to follow a kiss approach. |
I think that we will definitely need more information about the workflows - the Because we expect people to implement different tools to monitor workflows using this, I would advocate just sending everything and then people can choose what to use. Shouldn't be much data still and should be pretty easy to serialise into a JSON object. |
I see the rationale but, the full config can contains sensitive informations, eg. cloud security credentials, tokens and password as env vars, etc. therefore it would be required to strip all this information. |
hmm, yep, ok. How about just |
Hmm, not sure this is feasible but I agree that a restriction makes sense. I assume the same as Phil that an exclusion of the |
I agree, the information in |
+1 for |
How much work is required to get the parameters set in the main script? I suspect that this will be a fairly high priority for weblog to be picked up. |
Complexity index 9 out if 10. The problem here is the parameters in the
script are just variable assignments.
To evaluate them you need to run the full script.
The new modules syntax will introduce some changes that may allow to parse
them without running the full pipelines.
However it's not anytime soon.
…On Tue, Mar 19, 2019, 17:58 Phil Ewels ***@***.***> wrote:
How much work is required to get the parameters set in the main script? I
suspect that this will be a fairly high priority for weblog to be picked up.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#1076 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAx3SEhdJiriYMpxalJOnNqgC9FIzfHlks5vYRdHgaJpZM4b2i2B>
.
|
@ewels @johandahlberg I have appended the WorkflowMetadata content now to the payload. When do you want this to have included (event types)? |
Thanks for clarification @pditommaso ! @sven1103 I think once at the beginning of each workflow should suffice - or does it make more sense to send it upon successful (?) finalization of the job? Then we don't have to filter it afterward as it will only be there when a job finishes successfully... ? |
@pditommaso - but we're talking about a sending this weblog event after the workflow has started, so we're already running the full script here anyway? |
ouch, I was stuck in the other config issue! yes, here params are valid when the execution starts |
The metadata are complete on start, why they should be sent at the end ? |
For example: workflow completion date-time, duration, success flag, exit status |
+1 |
ok, implemented and works. Still need to gather the params |
Example output as appetizer
|
@sven1103 I agree with the others here that it would make sense to send it at the workflow start, and at workflow finishing (regardless of whether it was successful or not). |
@johandahlberg It will be now send when the workflow is started and when it is completed. In terms of failure, the For example docker daemon not running:
|
Awesome work @sven1103 😁 Once the params are added then I think we should have pretty much everything we'll need 👍 |
@sven1103 cool! Looks very useful. |
Ok, params sneak preview (nf-core/hlatyping):
|
@rsuchecki workflow metadata will be send as JSON payload soon as well: #1077 . This issue is solved imo so I will close it :) |
Manifest data in weblog message
The weblog message content should also provide the manifest information on workflow submit, such that it can be used for remote database logging.
Usage scenario
When you want to do remote logging of your workflows persistently in a database, and you need to relate the workflow manifest with the trace data.
Suggest implementation
Fetch the manifest object from the
Session
object as MapSession.manifest.toMap()
in theWebLogObserver
class. Add a propertymanifest
in the JSON message, if the manifest data is provided.The text was updated successfully, but these errors were encountered: