Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[438] Stdout/stderr streaming to blob storage #451

Merged
merged 9 commits into from
Oct 10, 2023
Merged

[438] Stdout/stderr streaming to blob storage #451

merged 9 commits into from
Oct 10, 2023

Conversation

giventocode
Copy link
Contributor

@giventocode giventocode commented Oct 7, 2023

Closes: #438

In this PR:

  • The standard output and error streams are redirected and sent to blob storage.
    • Each operation (download, executor, upload) generates log files for both standard output and error streams.
    • Data is written in blocks of 16KiB to an append blob.
  • Executor (the process that runs commands on the container) is launched as a separate process to enable consistent log streaming for all operations.
    • The CLI has a new command: exec.
    • The root command runs all commands/operations as sub-processes.
  • Logging format is changed. The log format is now simple instead of systemd.
  • Renamed 'BlockBlobApiHttpUtils' to 'BlobApiHttpUtils'

@giventocode giventocode marked this pull request as ready for review October 7, 2023 21:26
@giventocode giventocode requested a review from a team October 7, 2023 21:27
@giventocode giventocode changed the title [438] Sttdout/stderr streaming to blob storage [438] Stdout/stderr streaming to blob storage Oct 8, 2023
@giventocode giventocode merged commit cd78c00 into main Oct 10, 2023
8 checks passed
@giventocode giventocode deleted the ja-438 branch October 10, 2023 14:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[#148 Implement TES Task Runner] Improve node execution stdout and stderr publishing
2 participants