Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[#148 Implement TES Task Runner] Improve node execution stdout and stderr publishing #438

Closed
Tracked by #148
giventocode opened this issue Oct 2, 2023 · 0 comments · Fixed by #451
Closed
Tracked by #148
Assignees
Labels
Observability Enable users to view logs for how system is processing a task from start to end in one place

Comments

@giventocode
Copy link
Contributor

giventocode commented Oct 2, 2023

Problem:
Stdout/stderr files are only accessible after the task is finished. This makes it hard to troubleshoot long running tasks. Batch uploads these files using SAS tokens that the TES server generates beforehand. In Terra, these tokens might expire if the workflows take too long to execute

Solution:
Publish stdout and stderr data to Azure storage as they are being produced by the runner.

AC:

  • Create separate files for each operation: download-stdout.txt, execution-stdout.txt, upload-stdout.txt, download-stderr.txt, execution-stderr.txt, and upload-stderr.txt.
  • Use append blobs and be aware of their limits (4 MiB per block and 50,000 blocks maximum).
  • Upload operations must be asynchronous and not blocking.
  • The TES server must set the location of the logs and the transformation strategy to obtain the JIT SAS token.
  • Improve log format - the current md format is hard to read.
@ngambani ngambani changed the title Improve node execution stdout and stderr publishing [#148 Implement TES Task Runner] Improve node execution stdout and stderr publishing Oct 4, 2023
@giventocode giventocode self-assigned this Oct 5, 2023
@ngambani ngambani added the Observability Enable users to view logs for how system is processing a task from start to end in one place label Oct 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Observability Enable users to view logs for how system is processing a task from start to end in one place
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants