feat: make tracking OOM memory issues in schema worker easier#8258
Conversation
…e of an oom issue, write all the subgraphs as an artifact to s3 and add a reference to that on the trace
464c927 to
62d0fd0
Compare
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
| // If a composition error happens due to OOM we write the subgraphs to S3 | ||
| // This makes it easier to reproduce and debug the issue |
There was a problem hiding this comment.
Technically we could do that for every composition failure. But this would end up using a lot lot more s3 storage and we do not really have a cleanup strategy. Thus only storing the OOM ones for now seems kind okay in my opinion.
There was a problem hiding this comment.
In the future we want to store SDL on s3 exclusively anyways - as it is eating up a lot of space on postgres and would be more cost effective over there. Then this would no longer be needed.
| const span = trace.getActiveSpan(); | ||
| if (span) { |
There was a problem hiding this comment.
This behaviour is only enabled if tracing is enabled for now.
|
🐋 This PR was built and pushed to the following Docker images: Targets: Platforms: Image Tags: |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
1 participant
In case of an oom issue, write all the subgraphs as an artifact to s3 and add a reference to that on the trace.
I tested the flow locally by:
Setting low memory
Publishing a highe schema, causing an OOM for the worker thread
The trace is now populated with some reference:
And an artifact is written to our S3 bucket containing all the subgraphs involved in the composition:
