feat(buffer_object): Add resize method to BufferIO by Leahlijuan · Pull Request #57 · google/ml-flashpoint

Leahlijuan · 2026-02-23T17:37:47Z

Fixes #40

gemini-code-assist · 2026-02-23T17:38:13Z

Summary of Changes

Hello @Leahlijuan, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the BufferIO system by introducing the capability to dynamically resize buffer objects. This feature allows for greater flexibility in managing data stored in memory-mapped files, enabling the buffer's capacity to be adjusted after its initial creation. The implementation involves changes in both the Python BufferIO class and its underlying C++ BufferObject and memory-mapping utilities, ensuring robust and efficient resizing operations.

Highlights

Dynamic Buffer Resizing: Introduced a resize method to the BufferIO class in Python, allowing dynamic adjustment of buffer capacity for memory-mapped files.
C++ Backend Implementation: Implemented the core resize logic in the underlying C++ BufferObject, which handles the resizing of the memory-mapped file.
Memory-Mapped File Resizing Utility: Developed a new C++ helper function, resize_mmap, to manage the complex process of unmapping, truncating, and remapping files for resizing.
Comprehensive Testing: Added extensive unit tests across both Python and C++ components to ensure the resize functionality works correctly, preserves data, and handles various edge cases and error conditions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

src/ml_flashpoint/checkpoint_object_manager/buffer_io.py
- Added resize method to BufferIO class.
- Implemented logic to release and recreate memoryview and remap metadata after resizing.
- Included validity checks for write mode and minimum buffer size.
src/ml_flashpoint/checkpoint_object_manager/buffer_object/bindings.cpp
- Exposed BufferObject::resize method to Python via pybind11.
src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.cpp
- Implemented resize_mmap function to unmap, ftruncate, and remap memory-mapped files.
- Added error handling and logging for resize_mmap.
src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.h
- Declared resize_mmap function for memory-mapped file resizing.
src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_object.cpp
- Implemented BufferObject::resize method with checks for closed, read-only, or zero-capacity buffers.
- Integrated resize_mmap to perform the actual file resizing.
src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_object.h
- Declared resize method within the BufferObject class.
tests/checkpoint_object_manager/buffer_object/buffer_helper_test.cpp
- Added ResizeMmapSucceedsToLargerSize test.
- Added ResizeMmapSucceedsToSmallerSize test.
- Added ResizeMmapFailsOnInvalidFd test.
- Added ResizeMmapFailsOnFtruncateFailure test.
tests/checkpoint_object_manager/buffer_object/buffer_object_test.cpp
- Added ResizeSucceeds test.
- Added ResizePreservesData test.
- Added ResizeFailsOnClosedBuffer test.
- Added ResizeFailsOnReadOnlyBuffer test.
- Added ResizeFailsOnZeroCapacity test.
tests/checkpoint_object_manager/test_buffer_io.py
- Added TestResizeOperations class with tests for resize_increases_capacity, resize_updates_memoryview, resize_fails_on_closed_buffer, resize_fails_on_readonly_buffer, and resize_fails_on_zero_capacity.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

The pull request introduces a resize method to the BufferIO class and the underlying C++ BufferObject. The implementation is generally sound, correctly managing memory-mapped regions and metadata. I have identified a few areas for improvement: specifically, validating the new size upfront in BufferIO.resize to ensure atomicity and adhering to the repository's testing style guide by using structured comments (Given/When/Then) in the new test cases.

src/ml_flashpoint/checkpoint_object_manager/buffer_io.py

tests/checkpoint_object_manager/test_buffer_io.py

tests/checkpoint_object_manager/buffer_object/buffer_object_test.cpp

tests/checkpoint_object_manager/buffer_object/buffer_helper_test.cpp

src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.cpp

g-husam · 2026-02-23T18:26:45Z

src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.cpp

+  out_data_ptr = ptr;
+  out_data_size = new_size;
+
+  LOG(INFO) << "Successfully resized mmap to " << new_size;


make this debug

there is no LOG(DEBUG) for this library

ah true, can we use VLOG(DEBUG_LEVEL) then, that's the recommendation. can set constexpr DEBUG_LEVEL = 4 .

but for other methods, we use LOG(INFO) for similar logs, maybe we can keep it for now and modify the log in the future for all of them

src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.h

src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.cpp

src/ml_flashpoint/checkpoint_object_manager/buffer_io.py

tests/checkpoint_object_manager/test_buffer_io.py

tests/checkpoint_object_manager/buffer_object/buffer_helper_test.cpp

src/ml_flashpoint/checkpoint_object_manager/buffer_io.py

g-husam · 2026-02-25T15:21:17Z

src/ml_flashpoint/checkpoint_object_manager/buffer_object/buffer_helper.cpp

+  out_data_ptr = ptr;
+  out_data_size = new_size;
+
+  LOG(INFO) << "Successfully resized mmap to " << new_size;


ah true, can we use VLOG(DEBUG_LEVEL) then, that's the recommendation. can set constexpr DEBUG_LEVEL = 4 .

tests/checkpoint_object_manager/buffer_object/buffer_helper_test.cpp

github-actions · 2026-02-25T16:21:35Z

Python Code Coverage Summary

Package	Line Rate	Branch Rate	Health
src.ml_flashpoint	100%	100%	✔
src.ml_flashpoint.adapter	100%	100%	✔
src.ml_flashpoint.adapter.megatron	97%	94%	✔
src.ml_flashpoint.adapter.nemo	98%	94%	✔
src.ml_flashpoint.adapter.pytorch	99%	88%	✔
src.ml_flashpoint.checkpoint_object_manager	92%	91%	➖
src.ml_flashpoint.core	96%	92%	✔
src.ml_flashpoint.replication	81%	81%	❌
Summary	95% (2058 / 2170)	91% (473 / 520)	➖

Minimum allowed line rate is 90%

github-actions · 2026-02-25T16:21:38Z

C++ Code Coverage Summary

Package	Line Rate	Branch Rate	Health
src.ml_flashpoint.checkpoint_object_manager.buffer_object	93%	54%	✔
src.ml_flashpoint.checkpoint_object_manager.object_manager	70%	37%	❌
src.ml_flashpoint.replication.transfer_service	79%	40%	❌
Summary	81% (916 / 1126)	43% (687 / 1604)	➖

Minimum allowed line rate is 80%

Leahlijuan requested review from g-husam and kkkapu February 23, 2026 17:38

Leahlijuan mentioned this pull request Feb 23, 2026

perf: Implement BufferPool for efficient memory reuse #50

Closed

2 tasks

gemini-code-assist bot reviewed Feb 23, 2026

View reviewed changes

g-husam reviewed Feb 23, 2026

View reviewed changes

src/ml_flashpoint/checkpoint_object_manager/buffer_io.py Show resolved Hide resolved

Leahlijuan added 3 commits February 24, 2026 18:48

feat: Add resize method to BufferIO

ee6544b

Fix test issues

e7b0f6c

Resolve comments

9879a7e

Leahlijuan force-pushed the feat/bufferresize branch from b460288 to 9879a7e Compare February 24, 2026 18:48

Leahlijuan requested a review from g-husam February 25, 2026 15:14

g-husam approved these changes Feb 25, 2026

View reviewed changes

g-husam changed the title ~~feat: Add resize method to BufferIO~~ feat(buffer_object): Add resize method to BufferIO Feb 25, 2026

Leahlijuan and others added 2 commits February 25, 2026 16:11

Resolve comments

ccb2564

Merge branch 'main' into feat/bufferresize

629d4a5

Leahlijuan merged commit c45592d into main Feb 25, 2026
5 checks passed

Leahlijuan deleted the feat/bufferresize branch February 25, 2026 16:29

Conversation

Leahlijuan commented Feb 23, 2026

Uh oh!

gemini-code-assist bot commented Feb 23, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

g-husam Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Leahlijuan Feb 24, 2026

Choose a reason for hiding this comment

Uh oh!

g-husam Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Leahlijuan Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

g-husam Feb 25, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 25, 2026

Python Code Coverage Summary

Uh oh!

github-actions bot commented Feb 25, 2026

C++ Code Coverage Summary

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants