Remove full file load from publish step #3724
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This updates the logic in the `publishMavenCentralBundle` task so that it no longer loads the full contents of the file into memory. It does this by maintaining a buffer and then writing the contents of the buffer into the stream, which allows us to avoid materializing the full element into memory. This is to try and fix a Java heap memory error that we saw previously (see: https://github.com/FoundationDB/fdb-record-layer/actions/runs/19108664941/job/54605649248). While I was here, I also updated the boundary for the HTTP multi-part form from the current time millis to a UUID, in response to a comment from the previous PR (FoundationDB#3723 (comment)).
ScottDugas
approved these changes
Nov 5, 2025
ScottDugas
requested changes
Nov 5, 2025
build.gradle
Outdated
| // Stream the file in chunks to avoid loading it all into memory | ||
| bundleFile.withInputStream { input -> | ||
| connection.outputStream << input | ||
| byte[] buffer = new byte[8192] |
Collaborator
There was a problem hiding this comment.
Looking into it, I think the << does this already.... https://github.com/apache/groovy/blob/GROOVY_3_0_22/src/main/java/org/codehaus/groovy/runtime/IOGroovyMethods.java#L206
I think the actual problem is that if you are not streaming, HTTPUrlConnection will use a PosterOutputStream which is ByteArrayOutputStream.
I think you need to call setChunkedStreamingMode(int chunklen) or setFixedLengthStreamingMode(long contentLength)
I think we should be able to calculate the content length if you first convert the header and footer to byte[].
Collaborator
Author
There was a problem hiding this comment.
Okay, sure. I can give that a shot
ScottDugas
approved these changes
Nov 5, 2025
alecgrieser
added a commit
to alecgrieser/fdb-record-layer
that referenced
this pull request
Nov 6, 2025
This reverts commit f161cbc.
arnaud-lacurie
pushed a commit
that referenced
this pull request
Nov 7, 2025
This reverts #3723 and #3724 and returns us back to publishing via the nexus publishing API. The impetus for this is that the last approach got stuck in a publishing state after the upload (see: https://github.com/FoundationDB/fdb-record-layer/actions/runs/19117047380/job/54633288001, which succeeded, but for which we didn't get artifacts actually published despite the server upload succeeding). When we are in this mode, we expect the build to fail with a 400 error letting us know that we didn't upload anything. However, if we didn't `close` the staging repositories (which is the step that is failing), then legacy API would not be expected to upload anything to the central repo. So we need to continue to do that part even though we think it will fail.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This updates the logic in the
publishMavenCentralBundletask so that it no longer loads the full contents of the file into memory. It does this by maintaining a buffer and then writing the contents of the buffer into the stream, which allows us to avoid materializing the full element into memory. This is to try and fix a Java heap memory error that we saw previously (see: https://github.com/FoundationDB/fdb-record-layer/actions/runs/19108664941/job/54605649248).While I was here, I also updated the boundary for the HTTP multi-part form from the current time millis to a UUID, in response to a comment from the previous PR (#3723 (comment)).