From 2c2a324c9291bc473ba24fb3c967596bc918dc31 Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Wed, 5 Nov 2025 17:59:14 +0000 Subject: [PATCH 1/3] Remove full file load from publish step This updates the logic in the `publishMavenCentralBundle` task so that it no longer loads the full contents of the file into memory. It does this by maintaining a buffer and then writing the contents of the buffer into the stream, which allows us to avoid materializing the full element into memory. This is to try and fix a Java heap memory error that we saw previously (see: https://github.com/FoundationDB/fdb-record-layer/actions/runs/19108664941/job/54605649248). While I was here, I also updated the boundary for the HTTP multi-part form from the current time millis to a UUID, in response to a comment from the previous PR (https://github.com/FoundationDB/fdb-record-layer/pull/3723#discussion_r2495094985). --- build.gradle | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/build.gradle b/build.gradle index 430bdfd5f9..3d88dc8f21 100644 --- a/build.gradle +++ b/build.gradle @@ -373,24 +373,32 @@ if (Boolean.parseBoolean(publishBuild) && Boolean.parseBoolean(centralPublish)) connection.setDoOutput(true) connection.setRequestProperty('Authorization', "Bearer ${encodedCredentials}") - def boundary = "----GradleBoundary${System.currentTimeMillis()}" + def boundary = "----GradleBoundary${UUID.randomUUID().toString()}" connection.setRequestProperty('Content-Type', "multipart/form-data; boundary=${boundary}") - connection.outputStream.withWriter('UTF-8') { writer -> - writer.write("--${boundary}\r\n") - writer.write("Content-Disposition: form-data; name=\"bundle\"; filename=\"${bundleFile.name}\"\r\n") - writer.write("Content-Type: application/octet-stream\r\n") - writer.write("\r\n") - writer.flush() + // Use buffered output stream and write directly to avoid loading entire file into memory + connection.outputStream.withStream { output -> + def CRLF = "\r\n" - // Write the file bytes + // Write multipart headers + output.write("--${boundary}${CRLF}".getBytes('UTF-8')) + output.write("Content-Disposition: form-data; name=\"bundle\"; filename=\"${bundleFile.name}\"${CRLF}".getBytes('UTF-8')) + output.write("Content-Type: application/octet-stream${CRLF}".getBytes('UTF-8')) + output.write(CRLF.getBytes('UTF-8')) + + // Stream the file in chunks to avoid loading it all into memory bundleFile.withInputStream { input -> - connection.outputStream << input + byte[] buffer = new byte[8192] + int bytesRead + while ((bytesRead = input.read(buffer)) != -1) { + output.write(buffer, 0, bytesRead) + } } - writer.write("\r\n") - writer.write("--${boundary}--\r\n") - writer.flush() + // Write multipart closing boundary + output.write(CRLF.getBytes('UTF-8')) + output.write("--${boundary}--${CRLF}".getBytes('UTF-8')) + output.flush() } def responseCode = connection.responseCode From 62808ea010766b6d784747378f0a4f0f9cf708a0 Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Wed, 5 Nov 2025 18:04:09 +0000 Subject: [PATCH 2/3] remove release note for unpublished version --- docs/sphinx/source/ReleaseNotes.md | 33 ------------------------------ 1 file changed, 33 deletions(-) diff --git a/docs/sphinx/source/ReleaseNotes.md b/docs/sphinx/source/ReleaseNotes.md index 5f50f1ef76..521b0a62c8 100644 --- a/docs/sphinx/source/ReleaseNotes.md +++ b/docs/sphinx/source/ReleaseNotes.md @@ -7,39 +7,6 @@ As the [versioning guide](Versioning.md) details, it cannot always be determined ## 4.8 -### 4.8.10.0 - - -
- - -

Build/Test/Documentation/Style Improvements (click to expand)

- - - -* Replace nexus publishing plugin with direct call to central sonatype uploading URL - [PR #3723](https://github.com/FoundationDB/fdb-record-layer/pull/3723) -* Remove JReleaser publisher - [PR #3721](https://github.com/FoundationDB/fdb-record-layer/pull/3721) -* Increase heap size for JReleaser - [PR #3720](https://github.com/FoundationDB/fdb-record-layer/pull/3720) -* Clean up 4.8.4.0 and 4.8.7.0 release notes - [PR #3719](https://github.com/FoundationDB/fdb-record-layer/pull/3719) -* Upload the jreleaser trace logs & config - [PR #3718](https://github.com/FoundationDB/fdb-record-layer/pull/3718) -* Reduce test load to reduce flakiness - [PR #3712](https://github.com/FoundationDB/fdb-record-layer/pull/3712) -* Update the release plugin to use the central publishing API directly - [PR #3710](https://github.com/FoundationDB/fdb-record-layer/pull/3710) - -
- - -**[Full Changelog (4.8.6.0...4.8.10.0)](https://github.com/FoundationDB/fdb-record-layer/compare/4.8.6.0...4.8.10.0)** - -#### Mixed Mode Test Results - -Mixed mode testing run against the following previous versions: - -❌`4.6.4.0`, ❌`4.6.5.0`, ❌`4.7.1.0`, ❌`4.7.2.0`, ✅`4.7.3.0`, ✅`4.8.1.0`, ✅`4.8.2.0`, ✅`4.8.3.0`, ✅`4.8.5.0`, ✅`4.8.6.0` - -[See full test run](https://github.com/FoundationDB/fdb-record-layer/actions/runs/19108664941) - - - ### 4.8.6.0

New Features

From baaf4072a076e6c3f7eee2da688d8405f165c450 Mon Sep 17 00:00:00 2001 From: Alec Grieser Date: Wed, 5 Nov 2025 19:15:47 +0000 Subject: [PATCH 3/3] set useChunckedStreamingMode to enable multi-part chunking --- build.gradle | 13 +++++-------- 1 file changed, 5 insertions(+), 8 deletions(-) diff --git a/build.gradle b/build.gradle index 3d88dc8f21..f73d8167ec 100644 --- a/build.gradle +++ b/build.gradle @@ -376,6 +376,9 @@ if (Boolean.parseBoolean(publishBuild) && Boolean.parseBoolean(centralPublish)) def boundary = "----GradleBoundary${UUID.randomUUID().toString()}" connection.setRequestProperty('Content-Type', "multipart/form-data; boundary=${boundary}") + // Use a chunked streaming mode to break data up into smaller chunks to avoid needing to load everything into memory + connection.setChunkedStreamingMode(8192) + // Use buffered output stream and write directly to avoid loading entire file into memory connection.outputStream.withStream { output -> def CRLF = "\r\n" @@ -386,14 +389,8 @@ if (Boolean.parseBoolean(publishBuild) && Boolean.parseBoolean(centralPublish)) output.write("Content-Type: application/octet-stream${CRLF}".getBytes('UTF-8')) output.write(CRLF.getBytes('UTF-8')) - // Stream the file in chunks to avoid loading it all into memory - bundleFile.withInputStream { input -> - byte[] buffer = new byte[8192] - int bytesRead - while ((bytesRead = input.read(buffer)) != -1) { - output.write(buffer, 0, bytesRead) - } - } + // Next write the bundle data to the stream + bundleFile.withInputStream { InputStream input -> output << input } // Write multipart closing boundary output.write(CRLF.getBytes('UTF-8'))