Added IBucket.open_write()

[asuiu]

No description provided.

[coderabbitai]

Caution

Review failed

The pull request is closed.

📝 Walkthrough

Summary by CodeRabbit

• New Features
  • Added streaming write support via an open_write context manager across Filesystem, Memory, and MinIO buckets (atomic writes on FS; multipart streaming on MinIO). Includes synchronization to prevent concurrent writes to the same object.
  • Explicitly disallow writes on the cached immutable bucket.
• Refactor
  • Tightened validation across the bucket interface for stricter input checking.
• Tests
  • Expanded coverage for open_write, including large/chunked writes and Parquet round-trips on all supported backends.

Walkthrough

Adds a new streaming write API open_write across the bucket interface and implementations. IBucket introduces the abstract method and stricter validation base. FSBucket implements atomic temp-file writes. MemoryBucket implements in-memory sink. MinioBucket implements streaming multipart uploads via a background thread. CachedImmutableBucket exposes open_write but raises UnsupportedOperation. Tests expanded to cover open_write (incl. Parquet).

Changes

Cohort / File(s)	Summary
Core API + Sync Wrapper python/bucketbase/ibucket.py	IBucket now inherits from PydanticStrictValidated. Adds abstract/context-managed open_write(name) -> AbstractContextManager[BinaryIO]. AbstractAppendOnlySynchronizedBucket implements open_write with per-object locking and existence check before delegating to base bucket. Expanded typing/imports.
FS Atomic Writer python/bucketbase/fs_bucket.py	Implements open_write with temp-file creation, directory prep, yielding a writable binary stream, and atomic rename on context exit. Cleans up temp file on error. Adds contextmanager import.
In-Memory Writer python/bucketbase/memory_bucket.py	Implements open_write yielding a non-closing BytesIO buffer; on exit flushes, captures bytes, stores in _objects. Adds helper _NonClosingBytesIO and contextmanager typing/imports.
MinIO Streaming Multipart python/bucketbase/minio_bucket.py	Implements open_write using a bounded queue and background uploader thread feeding put_object(length=-1, part_size=MIN_PART_SIZE). Adds _QueueWriter/_QueueReader helpers, _uploader method, adjusts PART_SIZE usage, minor header access tweak.
Immutable Cache Stub python/bucketbase/cached_immutable_bucket.py	Adds public open_write that raises io.UnsupportedOperation, with contextmanager signature; adjusts imports.
Test Suite: Shared Tester python/tests/bucket_tester.py	Adds tester methods for open_write, including large chunked writes, gzip path, and PyArrow Parquet write/read validation. Imports pyarrow/pq.
Tests: Bucket Implementations python/tests/test_fs_bucket.py, python/tests/test_memory_bucket.py, python/tests/test_minio_bucket.py	Adds test_open_write and test_open_write_with_parquet for FSBucket, MemoryBucket, and MinioBucket, delegating to shared tester; minor import/path formatting updates.
Tests: Cached Immutable python/tests/test_cached_immutable_bucket.py	Updates MockMainBucket with open_write signature (Iterator[BinaryIO]) for interface parity; imports Iterator.
Tests: IBucket python/tests/test_ibucket.py	Adds open_write tests using MemoryBucket via IBucketTester; minor formatting of expected strings.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor Client
  participant IBucket as IBucket (interface)
  participant SyncWrap as AbstractAppendOnlySynchronizedBucket
  participant Impl as Base Bucket Impl

  Client->>IBucket: open_write(name)
  note over IBucket: Implemented by SyncWrap or concrete bucket
  IBucket->>SyncWrap: open_write(name)
  SyncWrap->>SyncWrap: acquire per-object lock
  SyncWrap->>Impl: exists(name)?
  alt exists
    SyncWrap-->>Client: raise FileExistsError
    SyncWrap->>SyncWrap: release lock
  else not exists
    SyncWrap->>Impl: open_write(name)
    Impl-->>SyncWrap: Context-managed BinaryIO
    SyncWrap-->>Client: yield sink
    Client-->>SyncWrap: close context
    SyncWrap->>SyncWrap: release lock
  end

Loading

sequenceDiagram
  autonumber
  actor Client
  participant MinioB as MinioBucket
  participant Writer as _QueueWriter
  participant Uploader as Uploader Thread
  participant MinIO as MinIO Server

  Client->>MinioB: open_write(name)
  MinioB->>Uploader: start(_uploader with _QueueReader)
  MinioB-->>Client: yield Writer (BinaryIO)
  loop while writing
    Client->>Writer: write(data chunk)
    Writer->>Uploader: enqueue chunk
  end
  Client->>Writer: close()
  Writer->>Uploader: signal EOF
  Uploader->>MinIO: put_object(length=-1, part_size=MIN_PART_SIZE, stream=reader)
  alt upload success
    Uploader-->>MinioB: ok
  else upload error
    Uploader-->>MinioB: propagate exception on exit
  end

Loading

sequenceDiagram
  autonumber
  actor Client
  participant FSB as FSBucket
  participant FS as Filesystem

  Client->>FSB: open_write(name)
  FSB->>FS: create tmp file in .bucketbase_tmp/
  FSB-->>Client: yield binary sink to tmp file
  Client-->>FSB: close context
  FSB->>FS: flush/close tmp
  FSB->>FS: atomic rename tmp -> final
  alt error
    FSB->>FS: remove tmp
    FSB-->>Client: raise
  end

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Poem

New lanes for bytes to race and write,
Temp to final, clean and tight.
Queues to rockets stream the load,
Parquet sings along the road.
Memory hums, MinIO flies—
One API, many skies. 🚀

Tip

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

---

📜 Recent review details

Configuration used: CodeRabbit UI
Review profile: ASSERTIVE
Plan: Pro

💡 Knowledge Base configuration:

• MCP integration is disabled by default for public repositories
• Jira integration is disabled by default for public repositories
• Linear integration is disabled by default for public repositories

You can enable these sources in your CodeRabbit configuration.

📥 Commits

Reviewing files that changed from the base of the PR and between b91be22 and ee3eb33.

⛔ Files ignored due to path filters (1)

• python/poetry.lock is excluded by !**/*.lock

📒 Files selected for processing (11)

• python/bucketbase/cached_immutable_bucket.py (2 hunks)
• python/bucketbase/fs_bucket.py (2 hunks)
• python/bucketbase/ibucket.py (4 hunks)
• python/bucketbase/memory_bucket.py (2 hunks)
• python/bucketbase/minio_bucket.py (4 hunks)
• python/tests/bucket_tester.py (6 hunks)
• python/tests/test_cached_immutable_bucket.py (2 hunks)
• python/tests/test_fs_bucket.py (6 hunks)
• python/tests/test_ibucket.py (6 hunks)
• python/tests/test_memory_bucket.py (1 hunks)
• python/tests/test_minio_bucket.py (2 hunks)

✨ Finishing Touches

• 📝 Generate Docstrings

🧪 Generate unit tests

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch py-mpu-stream

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.