Skip to content

Add Deref/DerefMut support for all memory abstractions#18

Merged
konard merged 26 commits intomodern-apifrom
deref-initiative
Apr 11, 2026
Merged

Add Deref/DerefMut support for all memory abstractions#18
konard merged 26 commits intomodern-apifrom
deref-initiative

Conversation

@uselessgoddess
Copy link
Copy Markdown
Member

@uselessgoddess uselessgoddess commented Apr 28, 2023

Summary

Implements Deref<Target=[T]> and DerefMut for all memory types (Alloc, FileMapped, Global, System, TempFile), enabling safe slice-based access through standard Rust indexing, slicing, and iteration — while preserving full backward compatibility with allocated()/allocated_mut().

What changed

  • Alloc<T, A> and FileMapped<T>: Added Deref/DerefMut impls; allocated()/allocated_mut() now delegate to deref
  • delegate_memory! macro: Generates Deref/DerefMut impls for Global, System, TempFile wrappers
  • 22 new tests in 4 categorized files covering indexing, slice methods, consistency, and all types

What users can now do

use platform_mem::{Global, RawMem};

let mut mem = Global::<u64>::new();
mem.grow_from_slice(&[3, 1, 4, 1, 5]).unwrap();

// Indexing and slicing — no unsafe, no .allocated() needed
assert_eq!(mem[0], 3);
assert_eq!(&mem[1..4], &[1, 4, 1]);

// All slice methods work via Deref
mem.sort();
assert_eq!(&mem[..], &[1, 1, 3, 4, 5]);

// Backward compatible — allocated() still works
assert_eq!(mem.allocated(), &[1, 1, 3, 4, 5]);

Backward compatibility

  • allocated()/allocated_mut() remain on the RawMem trait and work unchanged
  • ErasedMem and Box<dyn ErasedMem> continue to work unchanged
  • tests/coverage.rs is unmodified — all 72 existing coverage tests pass as-is
  • No existing features removed

Test organization

Deref tests are split into 4 categorized files in tests/:

  • deref_indexing.rs — element access, ranges ([i], [1..4], [2..], [..3], [..], [1..=3]), empty deref
  • deref_slice_methods.rslen(), is_empty(), iter(), iter_mut(), contains(), sort()
  • deref_consistency.rs — Deref/allocated() API equivalence, grow/shrink behavior
  • deref_all_types.rs — coverage for Global, System, TempFile, Alloc, FileMapped

Diff minimization

The src diff is purely additive where possible:

  • alloc.rs: +18 lines (Deref/DerefMut impls added, allocated() delegates to self)
  • file_mapped.rs: +18 lines (same pattern)
  • lib.rs: +15 lines (Deref/DerefMut added to macro, existing delegation unchanged)

Test plan

  • All existing tests pass (cargo test --all-features — 114 tests)
  • 22 new Deref-specific tests in 4 categorized files
  • Tests verify consistency between Deref and allocated() APIs
  • No new clippy warnings introduced
  • tests/coverage.rs unchanged from main

🤖 Generated with Claude Code

Fixes #18

uselessgoddess and others added 18 commits November 7, 2023 18:47
Start the way to modern api for doublets
Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #20
Create comprehensive README with:
- Project overview and features description
- Installation instructions with nightly Rust note
- Usage examples: Global allocator, FileMapped, TempFile
- Generic RawMem trait usage and type-erased memory examples
- API reference tables for RawMem trait and memory types
- Error handling documentation
- Related projects section

Fixes #20

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
The MaybeUninit::write_slice_cloned and MaybeUninit::slice_assume_init_mut
functions were removed in recent nightly Rust. These have been replaced with
inherent methods on [MaybeUninit<T>] slices that are now stable in Rust 1.93.0:

- MaybeUninit::write_slice_cloned(slice, src) -> slice.write_clone_of_slice(src)
- MaybeUninit::slice_assume_init_mut(slice) -> slice.assume_init_mut()

Also removed stabilized feature flags:
- unchecked_math (stable since 1.79.0)
- maybe_uninit_slice (stable since 1.93.0)
- inline_const (stable since 1.79.0)
- maybe_uninit_write_slice (stable since 1.93.0)

Added example scripts in experiments/ to verify README examples work correctly.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
The rustdoc warning for an unresolved link to `mem::zeroed` has been
fixed by changing it to `std::mem::zeroed` which is the correct path.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #22
- Add rust-toolchain.toml with channel = "nightly-2022-08-22"
- Restore nightly features removed in PR #21:
  - unchecked_math
  - maybe_uninit_slice
  - inline_const
  - maybe_uninit_write_slice
  - let_else
  - nonnull_slice_from_raw_parts
- Revert MaybeUninit API changes to use original methods:
  - MaybeUninit::write_slice_cloned()
  - MaybeUninit::slice_assume_init_mut()
- Make raw_mem module public for uninit function testing
- Add comprehensive test coverage (72 new tests) covering:
  - Alloc: creation, growth, shrinking, capacity overflow, debug
  - Global/System: new, default, grow/shrink, allocated_mut, size_hint
  - TempFile: new, new_in, grow, shrink, debug
  - FileMapped: from_path, grow, shrink, capacity overflow, debug
  - RawMem trait: all grow variants, shrink, size_hint
  - ErasedMem: Box<dyn ErasedMem>, variants with Sync/Send
  - Error: all variants display/debug
  - uninit module: fill, fill_with, edge cases
  - Thread safety: Send/Sync assertions
  - Drop behavior with Arc reference counting
  - Edge cases: zero elements, empty slices, large allocations

Fixes #22

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
- Add GitHub Actions workflow for CI/CD
  - Multi-platform testing (Ubuntu, macOS, Windows)
  - Automated version bumping and releases
  - Changelog fragment checking for PRs
- Add release automation scripts
  - bump-version.mjs, check-file-size.mjs
  - collect-changelog.mjs, create-github-release.mjs
  - get-bump-type.mjs, version-and-commit.mjs
- Add changelog.d/ for fragment-based changelog management
- Add CONTRIBUTING.md with development guidelines
- Add CHANGELOG.md for release tracking
- Add .gitignore with standard patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
The newer versions of tempfile (3.24+) have transitive dependencies
that require Rust 1.71+, which is incompatible with our pinned
nightly-2022-08-22 toolchain (rustc 1.65).

This fixes the Windows CI build failure caused by windows-link v0.2.1
requiring a newer rustc version.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Set nightly-2022-08-22 toolchain and add 100% test coverage
Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #24
- Update rust-toolchain.toml from nightly-2022-08-22 to nightly
- Update MaybeUninit slice API to use new stabilized method names:
  - slice_assume_init_mut() -> .assume_init_mut()
  - write_slice_cloned() -> .write_clone_of_slice()
- Remove stabilized features from feature list:
  - let_else, inline_const, nonnull_slice_from_raw_parts
  - unchecked_math, maybe_uninit_slice, maybe_uninit_write_slice
- Update dependencies: memmap2 0.9, tempfile 3, thiserror 2
- Update CI workflow to use dtolnay/rust-toolchain@nightly

Fixes #24

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit completes the migration from Rust nightly to stable Rust
by replacing all unstable features with stable alternatives:

- Use `allocator-api2` crate instead of `#![feature(allocator_api)]`
- Replace `fn_traits`/`unboxed_closures` with custom `FillFn` trait
- Replace `slice_ptr_get`/`ptr_as_uninit` with manual pointer arithmetic
- Replace `slice_range` with manual bounds checking
- Add stable implementations for MaybeUninit slice methods

Changes:
- rust-toolchain.toml: Change from nightly to stable
- Cargo.toml: Add allocator-api2 dependency
- src/lib.rs: Remove all #![feature(...)] declarations
- src/raw_mem.rs: Add FillFn trait, range_bounds_to_range, uninit helpers
- src/raw_place.rs: Use manual pointer arithmetic
- src/alloc.rs: Use allocator_api2::alloc::Allocator
- tests/coverage.rs: Use allocator_api2, add assert_matches macro
- experiments/*.rs: Remove feature flags
- .github/workflows/release.yml: Use stable toolchain
- README.md, CONTRIBUTING.md: Update documentation

Fixes #24

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
Migrate from Rust nightly to stable Rust
@konard
Copy link
Copy Markdown
Member

konard commented Dec 28, 2025

May be we somehow can make common trait that will support both allocated/allocated_mut and Deref, so we can actually compare them using a benchmark?

konard and others added 6 commits January 13, 2026 13:15
Adding CLAUDE.md with task information for AI processing.
This file will be removed when the task is complete.

Issue: #26
Implements GitHub issue #26: async memory access support.

- Add AsyncFileMem<T> type for async file-backed memory using tokio
- Add 'async' feature flag to enable async functionality
- Add 'io-uring' feature flag placeholder for future io_uring support
- Create comprehensive benchmarks comparing sync vs async operations
- Add async usage example in experiments/async_example.rs
- Update documentation with async installation and usage examples
- Add AsyncFileMem to the Memory Types table

AsyncFileMem provides:
- create/open/temp async constructors
- async grow/grow_filled/grow_zeroed/grow_from_slice/shrink operations
- get/set/as_slice/as_slice_mut for in-memory buffer access
- sync/flush for persisting changes to disk

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The io_uring feature was causing CI failures on macOS and Windows
because tokio-uring only compiles on Linux. Since the io_uring
implementation is not yet complete, remove the feature for now.
The async feature with tokio works cross-platform.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…d queue

Replace the buffer-based AsyncFileMem with a producer-consumer architecture:
- Single dedicated thread owns FileMapped (mmap) and processes commands
- Async callers send commands via mpsc channel, receive results via oneshot
- No additional memory buffer allocation — mmap is the sole data store
- Page faults only block the I/O thread, not async task threads

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add async memory operations with AsyncFileMem (mmap + I/O thread)
@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

Conflicts should be resolved.

We should support Deref in addition to existing logic for backward compatibility. And support if for all abstractions we have, so nothing breaks, but we have ability to have safer abstraction for users of the library.

So that as I understand should allow us to use regular indexes and slices and ranges, instead of unsafe raw pointers.

We need to ensure all changes are correct, consistent, validated, tested, logged and fully meet each and all discussed requirements (check issue description and all comments in issue and in pull request). Ensure all CI/CD checks pass.

@konard konard marked this pull request as draft April 10, 2026 22:28
@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

🤖 AI Work Session Started

Starting automated work session at 2026-04-10T22:28:32.646Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

Implement Deref<Target=[T]> and DerefMut for Alloc, FileMapped,
Global, System, and TempFile. This enables safe slice-based access
via indexing, slicing, ranges, iterators, and all slice methods,
while preserving allocated()/allocated_mut() for backward compatibility.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@konard konard force-pushed the deref-initiative branch from 0209e4d to 53f20af Compare April 10, 2026 22:34
@konard konard changed the title Deref Initiative Add Deref/DerefMut support for all memory abstractions Apr 10, 2026
@konard konard marked this pull request as ready for review April 10, 2026 22:35
@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $3.220898

📊 Context and tokens usage:

  • Context window: 103.1K / 1M (10%) input tokens, 17.6K / 128K (14%) output tokens

Total: (93.2K + 4.4M cached) input tokens, 17.6K output tokens, $3.220898 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus
  • Model: Claude Opus 4.6 (claude-opus-4-6)

📎 Log file uploaded as Gist (1376KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

Please try to organize code in such a way that git diff is smaller for easier review. And double check we put all tests out of src folder into tests folder. No previous features are removed or missing, only new Deref feature is added. And also we have 100% tests coverage and all tests file are split into categories. Not all tests in single file.

We need to ensure all changes are correct, consistent, validated, tested, logged and fully meet each and all discussed requirements (check issue description and all comments in issue and in pull request). Ensure all CI/CD checks pass.

@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

🤖 AI Work Session Started

Starting automated work session at 2026-04-10T22:38:52.268Z

The PR has been converted to draft mode while work is in progress.

This comment marks the beginning of an AI work session. Please wait for the session to finish, and provide your feedback.

@konard konard marked this pull request as draft April 10, 2026 22:38
…ests

- Restructure Deref/DerefMut impls as purely additive blocks (no modification
  of existing RawMem impl structure in alloc.rs and file_mapped.rs)
- Keep delegate_memory! macro's allocated()/allocated_mut() delegation unchanged
- Restore tests/coverage.rs to its original state (no deref tests mixed in)
- Split deref tests into 4 categorized files:
  - deref_indexing.rs: element access, ranges, empty deref
  - deref_slice_methods.rs: len, is_empty, iter, contains, sort
  - deref_consistency.rs: Deref/allocated() API equivalence, grow/shrink
  - deref_all_types.rs: Global, System, TempFile, Alloc, FileMapped coverage

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@konard konard marked this pull request as ready for review April 10, 2026 22:46
@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

🤖 Solution Draft Log

This log file contains the complete execution trace of the AI solution draft process.

💰 Cost: $2.235564

📊 Context and tokens usage:

  • Context window: 74.1K / 1M (7%) input tokens, 15.6K / 128K (12%) output tokens

Total: (63.6K + 2.9M cached) input tokens, 15.6K output tokens, $2.235564 cost

🤖 Models used:

  • Tool: Anthropic Claude Code
  • Requested: opus
  • Model: Claude Opus 4.6 (claude-opus-4-6)

📎 Log file uploaded as Gist (1052KB)


Now working session is ended, feel free to review and add any feedback on the solution draft.

@konard
Copy link
Copy Markdown
Member

konard commented Apr 10, 2026

✅ Ready to merge

This pull request is now ready to be merged:

  • All CI checks have passed
  • No merge conflicts
  • No pending changes

Monitored by hive-mind with --auto-restart-until-mergeable flag

@konard konard merged commit 1247941 into modern-api Apr 11, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants