Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2024 Pyarelal Knowles, MIT License
# Copyright (c) 2024-2025 Pyarelal Knowles, MIT License

cmake_minimum_required(VERSION 3.20)

Expand Down
50 changes: 40 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
# decodeless_mappedfile

[`decodeless`](https://github.com/decodeless) (previously no-decode) is a
collection of utility libraries for conveniently reading and writing files via
memory mapping. Components can be used individually or combined.
[`decodeless`](https://github.com/decodeless) is a collection of utility
libraries for conveniently reading and writing files via memory mapping.
Components can be used individually or combined.

`decodeless_mappedfile` is a small cross platform file mapping abstraction that
supports reserving virtual address space and growing a file mapping into it for
an exciting new way to write binary files. Also includes convenient read-only
and writable file mapping objects.
`decodeless_mappedfile` is a small cross platform file mapping abstraction. It
provides simple objects to read and write to existing files. However, the more
interesting feature is a resizable mapped file. It works by reserving some
virtual address space and growing a file mapping into it for a really convenient
and fast way to write binary files.

[decodeless_writer](https://github.com/decodeless/writer) conbines this and
[decodeless_allocator](https://github.com/decodeless/allocator) to conveniently
Expand All @@ -33,16 +34,16 @@ write complex data structures directly as binary data.

## Code Example

```
```cpp
// Memory map a read-only file
decodeless::file mapped(filename);
const int* numbers = reinterpret_cast<const int*>(mapped.data());
...
```

```
```cpp
// Create a file
size_t maxSize = 4096;
size_t maxSize = 1 << 30; // reserved virtual address; can be huge
decodeless::resizable_file file(filename, maxSize);
EXPECT_EQ(file.size(), 0);
EXPECT_EQ(file.data(), nullptr);
Expand All @@ -58,6 +59,35 @@ EXPECT_EQ(numbers[9], 9);
numbers[99] = 99;
```

## Building

This is a header-only C++20 library with CMake integration. Use any of:

- ```cmake
add_subdirectory(path/to/mappedfile)
```

- ```cmake
include(FetchContent)
FetchContent_Declare(
decodeless_mappedfile
GIT_REPOSITORY https://github.com/decodeless/mappedfile.git
GIT_TAG release_tag
GIT_SHALLOW TRUE
)
FetchContent_MakeAvailable(decodeless_mappedfile)
```

- ```cmake
find_package(decodeless_mappedfile REQUIRED CONFIG PATHS paths/to/search)
```

Then,

```cmake
target_link_libraries(myproject PRIVATE decodeless::mappedfile)
```

## Notes

- Windows implementation uses unofficial section API for `NtExtendSection` from
Expand Down
42 changes: 36 additions & 6 deletions include/decodeless/detail/mappedfile_linux.hpp
Original file line number Diff line number Diff line change
@@ -1,15 +1,16 @@
// Copyright (c) 2024 Pyarelal Knowles, MIT License
// Copyright (c) 2024-2025 Pyarelal Knowles, MIT License

#pragma once

#include <assert.h>
#include <decodeless/detail/mappedfile_common.hpp>
#include <errno.h>
#include <exception>
#include <fcntl.h>
#include <limits>
#include <optional>
#include <string.h>
#include <string>
#include <optional>
#include <sys/mman.h>
#include <sys/stat.h>
#include <sys/types.h>
Expand Down Expand Up @@ -128,14 +129,25 @@ class MemoryMap {
return static_cast<address_type>(static_cast<byte_type*>(m_address) + offset);
}
size_t size() const { return m_size; }
void sync(int flags = MS_SYNC | MS_INVALIDATE)
void sync(size_t offset, size_t size) const
requires Writable
{
assert(offset + size <= m_size);
size_t alignedOffset = offset & ~(pageSize() - 1);
size_t alignedSize = size + offset - alignedOffset;
void* offsetAddress = static_cast<void*>(
static_cast<std::byte*>(const_cast<void*>(m_address)) + alignedOffset);
if (msync(offsetAddress, alignedSize, MS_SYNC | MS_INVALIDATE) == -1)
throw LastError();
}
void sync() const
requires Writable
{
// ENOMEM "Cannot allocate memory" here likely means something remapped
// the range before this object went out of scope. I haven't found a
// good way to avoid this other than the user being careful to delete
// the object before remapping.
if (msync(const_cast<void*>(m_address), m_size, flags) == -1)
if (msync(const_cast<void*>(m_address), m_size, MS_SYNC | MS_INVALIDATE) == -1)
throw LastError();
}
void resize(size_t size) {
Expand Down Expand Up @@ -182,12 +194,22 @@ template <bool Writable>
class MappedFile {
public:
using data_type = std::conditional_t<Writable, void*, const void*>;
MappedFile(const fs::path& path, int mapFlags = MAP_PRIVATE)
MappedFile(const fs::path& path)
: m_file(path, Writable ? O_RDWR : O_RDONLY)
, m_mapped(nullptr, m_file.size(), mapFlags, m_file, 0) {}
, m_mapped(nullptr, m_file.size(), Writable ? MAP_SHARED : MAP_PRIVATE, m_file, 0) {}

data_type data() const { return m_mapped.address(); }
size_t size() const { return m_mapped.size(); }
void sync() const
requires Writable
{
m_mapped.sync();
}
void sync(size_t offset, size_t size) const
requires Writable
{
m_mapped.sync(offset, size);
}

private:
static constexpr int MapMemoryProtection = Writable ? PROT_READ | PROT_WRITE : PROT_READ;
Expand Down Expand Up @@ -218,6 +240,14 @@ class ResizableMappedFile {
if (size)
map(size);
}
void sync() const {
if (m_mapped)
m_mapped->sync();
}
void sync(size_t offset, size_t size) const {
if (m_mapped)
m_mapped->sync(offset, size);
}

// Override default move assignment so m_reserved outlives m_mapped
ResizableMappedFile& operator=(ResizableMappedFile&& other) noexcept {
Expand Down
Loading
Loading