Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

JSON logging support / configurable logger support #378

Closed
kislyuk opened this issue Apr 22, 2020 · 2 comments
Closed

JSON logging support / configurable logger support #378

kislyuk opened this issue Apr 22, 2020 · 2 comments

Comments

@kislyuk
Copy link
Contributor

kislyuk commented Apr 22, 2020

Some of the software we run inside WDLs supports JSON logging. Also, we want to be able to configure miniWDL itself to emit its logs as JSON. Combining the two, if a log line on stderr is inferred to be a JSON object, and miniWDL's logger is configured to emit JSON, then the log line should be deserialized and nested into the miniWDL logger's JSON, instead of ending up being double-serialized.

@mlin
Copy link
Collaborator

mlin commented Apr 22, 2020

The miniwdl internal logs flow through the standard Python logging framework including the structured logging "recipe". We should be able to hook up the python-json-logging handler including "custom object serialization."

The task standard error streams through a file ($run_directory/stderr.txt) which miniwdl tails when in verbose logging mode. Thinking ahead to production use cases where miniwdl's GIL may present a scalability bottleneck, we might want to look at publishing that filename for some other process to tail and parse/forward.

@mlin
Copy link
Collaborator

mlin commented Apr 28, 2020

@kislyuk #380 adds --log-json and a hook permitting a plugin to override how each line of the streaming stderr log is handled in verbose mode. I'd still like to explore if there's a reasonable solution that avoids miniwdl having to examine and parse the stderr stream contents, since that might cause a scalability bottleneck down the road.

@mlin mlin closed this as completed Jun 5, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants