Skip to content

Key Value Store Updates #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
naftaly merged 1 commit into from
Nov 16, 2025
Merged

Key Value Store Updates #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
178 changes: 111 additions & 67 deletions .github/workflows/benchmarks.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ jobs:
id: benchmark
run: |
echo "Running benchmarks..."
swift test 2>&1 | tee benchmark_output.txt
swift test --filter testPerformance 2>&1 | tee benchmark_output.txt

- name: Parse and format results
run: |
Expand All @@ -41,80 +41,124 @@ jobs:
content = f.read()

# Find all performance test results
pattern = r"testPerformance(\w+).*?average: ([\d.]+).*?relative standard deviation: ([\d.]+)%"
pattern = r"testPerformance(\w+).*?average: ([\d.]+)"
matches = re.findall(pattern, content)

# Start markdown output
output = ["## Performance Analysis\n"]
output.append("| Operation | Total Time | Operations | **Per-Op Time** | Main Thread Safe? |")
output.append("|-----------|-----------|-----------|-----------------|-------------------|")

# Define operation counts and special handling
op_counts = {
"Insert": 100,
"LookupHit": 100,
"LookupMiss": 100,
"Update": 100,
"Remove": 200, # 100 inserts + 100 removes
"Count": 100,
"Keys": 1,
"ToDictionary": 1,
"MixedOperations": 136 # 50+50+25+10+1+1
output = ["# 🚀 KeyValueStore Performance Benchmarks\n"]
output.append("*Optimized with double hashing, memcmp equality, and derived hash2*\n")

# Core operations section
output.append("## Core Operations (100 ops)\n")
output.append("| Operation | Time | Per-Op | Main Thread |")
output.append("|-----------|------|--------|-------------|")

core_ops = {
"Insert": ("Insert", 100),
"LookupHit": ("Lookup (hit)", 100),
"LookupMiss": ("Lookup (miss)", 100),
"Update": ("Update", 100),
"Remove": ("Remove", 200),
"Contains": ("Contains", 100),
}

op_names = {
"Insert": "Insert",
"LookupHit": "Lookup (hit)",
"LookupMiss": "Lookup (miss)",
"Update": "Update",
"Remove": "Remove",
"Count": "Count",
"Keys": "Keys (100 items)",
"ToDictionary": "toDictionary (100)",
"MixedOperations": "Mixed Operations"
for test_name, avg_time in matches:
if test_name in core_ops:
avg_time_f = float(avg_time)
op_name, ops = core_ops[test_name]

total_ms = avg_time_f * 1000
per_op_us = (avg_time_f * 1_000_000) / ops

if per_op_us < 1:
per_op = "<1 μs"
elif per_op_us < 1000:
per_op = f"{per_op_us:.1f} μs"
else:
per_op = f"{per_op_us/1000:.2f} ms"

if total_ms < 10:
status = "✅ Excellent"
elif total_ms < 50:
status = "✅ Good"
elif total_ms < 100:
status = "⚠️ OK"
else:
status = "❌ Review"

output.append(f"| {op_name} | {total_ms:.1f}ms | {per_op} | {status} |")

# Load factor performance
output.append("\n## Load Factor Performance (10,000 lookups)\n")
output.append("| Load % | Time | Degradation | Status |")
output.append("|--------|------|-------------|--------|")

load_factors = {
"LoadFactor25Percent": ("25%", None),
"LoadFactor50Percent": ("50%", None),
"LoadFactor75Percent": ("75%", None),
"LoadFactor90Percent": ("90%", None),
"LoadFactor99Percent": ("99%", None),
}

for test_name, avg_time, std_dev in matches:
avg_time_f = float(avg_time)
total_time = f"{avg_time_f * 1000:.1f}ms"

ops = op_counts.get(test_name, 1)
op_name = op_names.get(test_name, test_name)

# Calculate per-operation time
per_op_us = (avg_time_f * 1_000_000) / ops

if per_op_us < 1:
per_op = f"**<1 μs**"
elif per_op_us < 1000:
per_op = f"**{per_op_us:.0f} μs**"
else:
per_op = f"**{per_op_us/1000:.1f} ms**"

# Main thread safety (16.67ms budget for 60fps)
if per_op_us < 100:
main_thread = "✅ Excellent"
elif per_op_us < 1000:
main_thread = "✅ Good"
elif per_op_us < 5000:
main_thread = "⚠️ OK"
else:
main_thread = "❌ Slow"

output.append(f"| {op_name} | {total_time} | {ops} | {per_op} | {main_thread} |")

# Add main thread budget info
output.append("")
output.append("### Main Thread Budget")
output.append("For smooth 60fps: **16.67ms per frame**")
output.append("For ProMotion 120fps: **8.33ms per frame**")
output.append("")

# Add footer
baseline = None
for test_name, avg_time in matches:
if test_name in load_factors:
avg_time_f = float(avg_time)
load_name = load_factors[test_name][0]

if baseline is None:
baseline = avg_time_f
degradation = "baseline"
else:
ratio = avg_time_f / baseline
degradation = f"{ratio:.1f}x"

total_ms = avg_time_f * 1000

if avg_time_f < 0.050:
status = "✅ Excellent"
elif avg_time_f < 0.100:
status = "✅ Good"
elif avg_time_f < 0.150:
status = "⚠️ OK"
else:
status = "❌ Slow"

output.append(f"| {load_name} | {total_ms:.0f}ms | {degradation} | {status} |")

# Key length impact
output.append("\n## Key Length Impact (100 ops)\n")
output.append("| Key Length | Time | Per-Op |")
output.append("|------------|------|--------|")

key_tests = {
"ShortKeys": "Short (2-3 chars)",
"MediumKeys": "Medium (~25 chars)",
"LongKeys": "Long (64 chars)",
}

for test_name, avg_time in matches:
if test_name in key_tests:
avg_time_f = float(avg_time)
key_name = key_tests[test_name]

total_ms = avg_time_f * 1000
per_op_us = (avg_time_f * 1_000_000) / 100

output.append(f"| {key_name} | {total_ms:.1f}ms | {per_op_us:.1f} μs |")

# Main thread budget
output.append("\n## Main Thread Guidelines")
output.append("- ✅ **Excellent**: <10ms - Perfect for UI interactions")
output.append("- ✅ **Good**: 10-50ms - Acceptable for most operations")
output.append("- ⚠️ **OK**: 50-100ms - Use with caution on main thread")
output.append("- ❌ **Review**: >100ms - Consider background thread")
output.append("\n*Target: 16.67ms per frame @ 60fps, 8.33ms @ 120fps*")

# Summary
total_tests = len(re.findall(r"Test Case.*passed", content))
output.append(f"**Total tests:** {total_tests} passed")
output.append("")
output.append(f"_Generated on {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}_")
output.append(f"\n---\n**Total tests:** {total_tests} passed | _Generated {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}_")

# Write to file
with open('benchmark_results.md', 'w') as f:
Expand Down
62 changes: 37 additions & 25 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,16 +7,17 @@

MemoryMap is a Swift utility class designed for efficient persistence and crash-resilient storage of Plain Old Data (POD) structs using memory-mapped files. It provides thread-safe access to the stored data, ensuring integrity and performance for applications requiring low-latency storage solutions.

## 🌟 Features
## Features

- **Memory-mapped file support**: Back a POD struct with a memory-mapped file for direct memory access
- **Thread-safe access**: Read and write operations protected by NSLock
- **Crash resilience**: Changes immediately reflected in the memory-mapped file
- **Data integrity validation**: File validation using magic numbers
- **KeyValueStore**: High-performance hash table with Dictionary-like API (30-50μs per operation)
- **KeyValueStore**: High-performance hash table with Dictionary-like API (10-20μs per operation)
- **Main-thread safe**: All operations optimized for UI thread usage
- **Double hashing**: Eliminates clustering for consistent performance even at high load factors

## 🔧 Installation
## Installation

To get started with MemoryMap, integrate it directly into your project:

Expand All @@ -25,7 +26,7 @@ To get started with MemoryMap, integrate it directly into your project:
3. Specify the version or branch you want to use.
4. Follow the prompts to complete the integration.

## 🚀 Usage
## Usage

### MemoryMap - Direct POD Storage

Expand Down Expand Up @@ -69,10 +70,10 @@ store["user:123"] = UserData(lastSeen: Date().timeIntervalSince1970, loginCount:
// Read with default value
let data = store["user:456", default: UserData(lastSeen: 0, loginCount: 0)]

// Update and get old value
let oldData = try store.updateValue(
// Explicit error handling
try store.setValue(
UserData(lastSeen: Date().timeIntervalSince1970, loginCount: 5),
forKey: "user:123"
for: "user:123"
)

// Iterate over keys
Expand All @@ -82,29 +83,38 @@ for key in store.keys {
}
}

// Compact to remove tombstones and improve performance
store.compact()

// Convert to Dictionary for advanced operations
let dict = store.toDictionary()
let dict = store.dictionaryRepresentation()
```

## ⚡ Performance
## Performance

KeyValueStore uses **double hashing** with optimized comparisons for excellent main-thread performance:

KeyValueStore is optimized for main-thread usage:
| Operation | Time (100 ops) | Per-Op | Main Thread |
|-----------|----------------|--------|-------------|
| Insert | 1.0ms | 10 μs | ✅ Excellent |
| Lookup (hit) | 1.0ms | 10 μs | ✅ Excellent |
| Lookup (miss) | 2.0ms | 20 μs | ✅ Excellent |
| Update | 2.0ms | 20 μs | ✅ Excellent |
| Remove | 3.0ms | 15 μs | ✅ Excellent |

| Operation | Per-Op Time | Main Thread Safe? |
|-----------|-------------|-------------------|
| Insert | 40 μs | ✅ Excellent |
| Lookup | 30 μs | ✅ Excellent |
| Update | 40 μs | ✅ Excellent |
| Remove | 50 μs | ✅ Excellent |
| Keys (100 items) | 1 ms | ✅ Good |
**Load Factor Performance:**
- 25% load: 17ms (baseline)
- 50% load: 35ms (2.1x)
- 75% load: 57ms (3.4x)
- 99% load: 106ms (6.2x) ⚠️

**Main Thread Budget:**
- 60fps: 16.67ms per frame
- 120fps: 8.33ms per frame

All operations are well within budget for smooth UI performance.
All operations are well within budget for smooth UI performance. Even at 99% capacity, performance remains acceptable for main thread usage.

## 🛠️ Development
## Development

### Code Formatting

Expand All @@ -126,21 +136,23 @@ swiftformat Sources/ Tests/
# Run all tests
swift test

# Run with AddressSanitizer
xcodebuild test -scheme MemoryMap -enableAddressSanitizer YES
# Run with sanitizers
swift test --sanitize thread # Thread Sanitizer
swift test --sanitize address # Address Sanitizer
swift test --sanitize undefined # Undefined Behavior Sanitizer

# Run with ThreadSanitizer
xcodebuild test -scheme MemoryMap -enableThreadSanitizer YES
# Run only performance benchmarks
swift test --filter testPerformance
```

### Benchmarks

Performance benchmarks run automatically on PRs and pushes to main via GitHub Actions. Results are posted as comments on PRs.

## 👋 Contributing
## Contributing

Got ideas on how to make MemoryMap even better? We'd love to hear from you! Feel free to fork the repo, push your changes, and open a pull request. You can also open an issue if you run into bugs or have feature suggestions.

## 📄 License
## License

MemoryMap is proudly open-sourced under the MIT License. Dive into the LICENSE file for more details.
Loading