-
Notifications
You must be signed in to change notification settings - Fork 0
English wiki
amcx is a binary file format (.amcx) designed to store AI chat memory in a compressed, structured, and verifiable way.
0.3.4
When an AI app needs to remember long conversations, the naive approach is to keep everything in plain text or JSON. This wastes disk space and forces the AI to load the entire history on every response.
amcx solves this by:
- Compressing chat history 60-70% using zlib/lzma
- Dividing memory into chunks so only relevant parts are loaded
- Verifying integrity with CRC32 and SHA-1 checksums
- Recovering corrupted chunks automatically with XOR recovery blocks
User sends message
↓
Developer's app calls SmartMemory.append()
↓
amcx compresses and saves to .amcx file
↓
User asks something from early in the conversation
↓
SmartMemory.search() loads ONLY the relevant chunk
↓
Chunk is decompressed in RAM, used, then discarded
↓
No temporary files left on disk
The end user never knows .amcx exists.
Try using amcl, a lighter version suitable for small businesses or projects with limited resources
Try using Add-ons, a native addons on c++
from amcx import SmartMemory
memory = SmartMemory("chat.amcx", use_mirror=False)
memory.append("user: Hello, tell me about the world")
memory.append("ai: The world is vast and full of mysteries...")
results = memory.search("world")
print(results)That's all the developer needs to write. Everything else is handled automatically.
- Python 3.10 or higher
- No external dependencies (uses stdlib only)
pip install amcxfrom amcx import SmartMemory
# Create a memory file
memory = SmartMemory("chat.amcx", use_mirror=False)
# Save messages
memory.append("user: Hi, what is the capital of France?")
memory.append("ai: The capital of France is Paris.")
memory.append("user: And what about Germany?")
memory.append("ai: The capital of Germany is Berlin.")
# Messages are automatically saved and compressedfrom amcx import SmartMemory
memory = SmartMemory("chat.amcx")
# Search for relevant context
results = memory.search("France")
for r in results:
print(r)
# Output: "user: Hi, what is the capital of France?"
# "ai: The capital of France is Paris."
# Get the most recent messages
recent = memory.get_recent(3)
for r in recent:
print(r)from amcx import SmartMemory
class MyChatBot:
def __init__(self):
# The user never knows this exists
self.memory = SmartMemory(".chat_history.amcx", use_mirror=False)
def respond(self, user_message: str) -> str:
# Save user message
self.memory.append(f"user: {user_message}")
# Load relevant context (decompresses only needed chunks)
context = self.memory.search(user_message, max_results=3)
# Generate response using your AI API here
response = your_ai_api(user_message, context)
# Save response
self.memory.append(f"ai: {response}")
return response
bot = MyChatBot()
print(bot.respond("Hello!"))memory = SmartMemory(
path="chat.amcx", # path to the .amcx file
use_mirror=False, # True = SHA-1 checksums + more space
use_recovery=False, # True = XOR recovery blocks
auto_chunk_size=2000, # characters before creating a new chunk
old_chunk_days=7, # days before switching to lzma compression
)| Option | Default | Description |
|---|---|---|
use_mirror |
False |
Embed SHA-1 checksums for integrity verification |
use_recovery |
False |
Add XOR recovery blocks for auto-repair |
auto_chunk_size |
2000 |
Max characters per chunk before splitting |
old_chunk_days |
7 |
Days until a chunk is compressed with lzma |
SmartMemory is the main high-level interface for amcx. It handles compression, chunking, integrity verification, and recovery automatically. The developer only needs to call simple methods.
from amcx import SmartMemorySmartMemory(
path: str,
use_mirror: bool = False,
use_recovery: bool = False,
auto_chunk_size: int = 2000,
old_chunk_days: int = 7,
)| Parameter | Type | Default | Description |
|---|---|---|---|
path |
str |
required | Path to the .amcx file |
use_mirror |
bool |
False |
Embed SHA-1 checksums per chunk |
use_recovery |
bool |
False |
Add XOR recovery blocks |
auto_chunk_size |
int |
2000 |
Characters before auto-saving a new chunk |
old_chunk_days |
int |
7 |
Days before compressing with lzma instead of zlib |
Adds a message to memory. Auto-saves when enough content accumulates.
memory.append("user: Hello!")
memory.append("ai: Hi, how can I help?")- Compression is chosen automatically (zlib for recent, lzma for old)
- No temporary files are created
- Thread-safe for single-process use
Searches for relevant messages. Only decompresses chunks that match.
results = memory.search("France", max_results=3)
for r in results:
print(r)Returns: List[str] — list of matching messages
| Parameter | Type | Default | Description |
|---|---|---|---|
query |
str |
required | Text to search for |
max_results |
int |
5 |
Maximum number of results |
Returns the N most recent messages.
recent = memory.get_recent(10)Returns: List[str] — most recent messages, newest first
Forces all pending messages to be saved to disk. Called automatically, but can be called manually.
memory.flush()Checks if the file is intact. Returns True if everything is fine.
if not memory.verify_integrity():
print("File may be corrupted")Returns: bool
Attempts to repair corrupted chunks using mirror or XOR recovery blocks.
fixed = memory.repair()
if fixed:
print("Repaired successfully")Returns: bool — True if something was repaired
Returns the file size in bytes.
size = memory.size_on_disk()
print(f"{size / 1024:.1f} KB")Returns: int
Returns total number of messages (saved + pending).
total = memory.count_messages()
print(f"Total messages: {total}")Returns: int
SmartMemory supports the with statement. It automatically calls flush() on exit.
with SmartMemory("chat.amcx") as memory:
memory.append("user: Hello")
memory.append("ai: Hi!")
# flush() is called automatically here| Configuration | File size (100 KB input) | Notes |
|---|---|---|
use_mirror=False |
~35 KB | Maximum savings |
use_mirror=True |
~50 KB | + SHA-1 verification |
use_mirror=True, use_recovery=True |
~60 KB | + XOR auto-repair |
from amcx import SmartMemory
class AIAssistant:
def __init__(self, session_id: str):
self.memory = SmartMemory(
f".sessions/{session_id}.amcx",
use_mirror=False,
auto_chunk_size=3000,
)
def chat(self, user_input: str) -> str:
# Save input
self.memory.append(f"user: {user_input}")
# Get context
context = self.memory.search(user_input, max_results=5)
recent = self.memory.get_recent(10)
# Call your AI API with context + recent messages
response = call_ai_api(user_input, context, recent)
# Save response
self.memory.append(f"ai: {response}")
return response
def stats(self):
return {
"messages": self.memory.count_messages(),
"size_kb": self.memory.size_on_disk() / 1024,
"healthy": self.memory.verify_integrity(),
}The .amcx format is a custom binary file designed for efficient storage and retrieval of AI chat memory.
[ HEADER 32b ] [ INDEX ] [ CHUNKS... ] [ RECOVERY BLOCK* ] [ MIRROR BLOCK* ]
* optional — only present if enabled
Everything is stored in a single file. No sidecars, no temporary files.
The first 4 bytes of every .amcx file are:
41 4D 43 00
A M C \0
Any program can read these 4 bytes to confirm the file is a valid .amcx before parsing anything else.
| Offset | Size | Field | Description |
|---|---|---|---|
0x00 |
4b | Magic | AMC\0 |
0x04 |
1b | Version major | Currently 0
|
0x05 |
1b | Version minor | Currently 3
|
0x06 |
4b | Num chunks | Total number of chunks |
0x0A |
8b | Created at | Unix timestamp |
0x12 |
4b | Index offset | Byte offset where index starts |
0x16 |
4b | Index size | Size of the index block in bytes |
0x1A |
2b | Flags | Bitfield (see below) |
0x1C |
4b | CRC32 | CRC32 of the first 28 bytes |
| Bit | Constant | Description |
|---|---|---|
| 0 | FLAG_COMPRESSED |
At least one chunk is compressed |
| 1 | FLAG_ENCRYPTED |
Reserved for future encryption |
| 2 | FLAG_READONLY |
File is read-only |
| 3 | FLAG_HAS_ACTIVE |
There is an active chunk |
| 4 | FLAG_HAS_ASSETS |
Reserved for future assets |
The index comes right after the header. It contains one entry per chunk (96 bytes each).
| Offset | Size | Field | Description |
|---|---|---|---|
0x00 |
4b | Chunk ID | Unique integer ID |
0x04 |
4b | Offset | Byte offset of the chunk in file |
0x08 |
4b | Size compressed | Compressed size in bytes |
0x0C |
4b | Size original | Original size before compression |
0x10 |
2b | Chunk type | See chunk types below |
0x12 |
1b | Algorithm | Compression algorithm |
0x13 |
1b | Reserved | Padding |
0x14 |
8b | Timestamp | Unix timestamp of the chunk |
0x1C |
4b | CRC32 | CRC32 of compressed data |
0x20 |
64b | Summary | UTF-8 text, null-padded |
| Value | Constant | Description |
|---|---|---|
0x00 |
CHUNK_LORE |
World/lore/rules |
0x01 |
CHUNK_CHARACTER |
Character data |
0x02 |
CHUNK_EVENT |
Narrative event |
0x03 |
CHUNK_ACTIVE |
Current active chunk |
0x04 |
CHUNK_GENERIC |
Generic content |
| Value | Constant | Algorithm | Best for |
|---|---|---|---|
0x00 |
COMPRESS_NONE |
None | Debug / tiny chunks |
0x01 |
COMPRESS_ZLIB |
zlib | Active/recent chunks |
0x02 |
COMPRESS_LZMA |
lzma | Old/archived chunks |
Each chunk is stored as:
[ 4 bytes: compressed size ] [ N bytes: compressed data ]
The index tells you exactly where each chunk starts, so reading one chunk never requires reading others.
Magic: AMCXR\0 (6 bytes)
XOR parity blocks inspired by WinRAR's recovery records. Chunks are grouped (default: 3 per group). Each group has a parity block = XOR of all chunks in the group. If one chunk is corrupted, it can be reconstructed using the others + parity.
AMCXR\0
uint32 num_groups
[ per group: ]
uint32 group_id
uint32 num_chunk_ids
[uint32 chunk_id ...]
uint32 parity_size
[bytes parity_data]
Magic: AMCXM\0 (6 bytes)
SHA-1 checksums for each chunk stored inside the file itself.
AMCXM\0
uint32 version
uint32 num_entries
uint64 timestamp
[ per chunk: ]
uint32 chunk_id
uint32 size_original
20 bytes SHA-1 (raw)
uint8 summary_len
N bytes summary (UTF-8)
The reader uses rfind() to locate the Recovery and Mirror blocks at the end of the file. This means:
- Load header → know where index is
- Load index → know where every chunk is
- Request chunk N → seek directly to its offset, read only those bytes
- Verify CRC32 → decompress → return
No scanning, no loading unnecessary data.
amcx includes two optional data protection systems, both embedded inside the .amcx file itself — no sidecar files.
| System | What it does | Space cost |
|---|---|---|
| Mirror | Stores SHA-1 per chunk to detect modifications or corruption | ~24 bytes × chunks |
| Recovery | Stores XOR parity blocks to reconstruct corrupted chunks | ~size of one chunk per group |
from amcx import SmartMemory
memory = SmartMemory(
"chat.amcx",
use_mirror=True, # SHA-1 checksums
use_recovery=True, # XOR recovery blocks
)Or with the low-level API:
from amcx import AMCXWriter, MirrorMode
writer = AMCXWriter(
mirror=MirrorMode.AUTO, # embed mirror on every save
recovery=True, # append XOR blocks
recovery_group=3, # chunks per parity group
)When use_mirror=True, a AMCXM\0 block is appended to the end of the file after every save. This block contains:
- A SHA-1 hash of each chunk's original (uncompressed) content
- The timestamp when the mirror was created
from amcx import AMCXMirror
status = AMCXMirror.verify("chat.amcx")
print(status.report())
# with the native SHA-1 accelerator
status = AMCXMirror.verify("chat.amcx", accelerator_path="./amcx_sha1.so")Example output:
Mirror: ✓ exists
✓ Mirror up to date
ID Status Summary
----------------------------------------------------
0 ✓ ok user: Hello, tell me about...
1 ✓ ok ai: The world is vast...
2 ✓ ok user: What about black holes?
✓ All good.
| Status | Meaning |
|---|---|
ok |
SHA-1 matches |
modified |
SHA-1 changed — chunk was edited |
missing_mirror |
Chunk in file but not in mirror |
missing_orig |
Chunk in mirror but not in file |
outdated |
Mirror is older than the file |
AMCXMirror.update("chat.amcx")
# with the native SHA-1 accelerator
AMCXMirror.update("chat.amcx", accelerator_path="./amcx_sha1.so")XOR parity inspired by WinRAR's recovery records. Chunks are divided into groups of N (default 3). For each group, a parity block is computed:
parity = chunk_0 XOR chunk_1 XOR chunk_2
If chunk_1 gets corrupted:
chunk_1 = parity XOR chunk_0 XOR chunk_2
from amcx import AMCXRecovery
can = AMCXRecovery.can_recover("chat.amcx", damaged_chunk_id=2)
print(can) # True or Falserecovered_bytes = AMCXRecovery.recover_chunk("chat.amcx", damaged_chunk_id=2)
text = recovered_bytes.decode("utf-8")
print(text)
# with the native XOR accelerator
recovered_bytes = AMCXRecovery.recover_chunk("chat.amcx", damaged_chunk_id=2, accelerator_path="./amcx_xor.so")- Only one chunk per group can be recovered
- If two chunks in the same group are corrupted, recovery fails
- Group size is set at write time and cannot be changed later
The MirrorMode enum controls when the mirror is updated:
| Mode | Behavior |
|---|---|
MirrorMode.NONE |
No mirror embedded |
MirrorMode.MANUAL |
Only when you call embed_mirror()
|
MirrorMode.AUTO |
Updated automatically on every save()
|
from amcx import AMCXWriter, MirrorMode
# Manual mode
writer = AMCXWriter(mirror=MirrorMode.MANUAL)
writer.save("memory.amcx")
writer.embed_mirror("memory.amcx") # called explicitly
# Auto mode
writer = AMCXWriter(mirror=MirrorMode.AUTO)
writer.save("memory.amcx") # mirror embedded automaticallyFor a 100 KB chat history:
| Configuration | File size |
|---|---|
| No mirror, no recovery | ~35 KB |
| Mirror only | ~50 KB |
| Mirror + recovery | ~60 KB |
The extra space buys you automatic corruption detection and repair.
For advanced use cases where you need full control over the .amcx file.
Creates and writes .amcx files.
from amcx import AMCXWriter, ChunkEntry, MirrorMode
from amcx import CHUNK_LORE, CHUNK_CHARACTER, CHUNK_EVENT, CHUNK_ACTIVE, CHUNK_GENERIC
from amcx import COMPRESS_NONE, COMPRESS_ZLIB, COMPRESS_LZMAAMCXWriter(
mirror: MirrorMode = MirrorMode.NONE,
recovery: bool = False,
recovery_group: int = 3,
)writer = AMCXWriter()
# Text shortcut
writer.add_text_chunk(
chunk_id=0,
chunk_type=CHUNK_LORE,
summary="The world",
text="The world is a dark place...",
algorithm=COMPRESS_LZMA,
)
# Raw bytes
writer.add_chunk(ChunkEntry(
chunk_id=1,
chunk_type=CHUNK_ACTIVE,
summary="Current session",
content=b"raw bytes here",
algorithm=COMPRESS_ZLIB,
))# Save to file
writer.save("memory.amcx")
# Get as bytes (no file)
data = writer.to_bytes()Reads .amcx files. Loads the index on open, chunks only on demand.
from amcx import AMCXReader# Context manager (recommended)
with AMCXReader("memory.amcx") as reader:
...
# Manual
reader = AMCXReader("memory.amcx")
reader.close()with AMCXReader("memory.amcx") as reader:
entries = reader.list_chunks()
for entry in entries:
print(f"[{entry.chunk_id}] {entry.summary} ({entry.algorithm_name}, {entry.size_original}b)")with AMCXReader("memory.amcx") as reader:
# Read as bytes
data = reader.read_chunk(0)
# Read as text
text = reader.read_chunk_text(0)
# Read active chunk
active = reader.read_active_chunk()with AMCXReader("memory.amcx") as reader:
print(reader.header.version_str) # "0.3"
print(reader.header.num_chunks) # 5
print(reader.header.has_active_chunk) # True/False
print(reader.summary()) # human-readable index| Constant | Value | Use for |
|---|---|---|
CHUNK_LORE |
0 |
World building, rules, setting |
CHUNK_CHARACTER |
1 |
Character descriptions |
CHUNK_EVENT |
2 |
Past events, narrative beats |
CHUNK_ACTIVE |
3 |
Current session (most recent) |
CHUNK_GENERIC |
4 |
Anything else |
| Constant | Algorithm | Speed | Ratio | Use for |
|---|---|---|---|---|
COMPRESS_NONE |
None | Fastest | 0% | Debug |
COMPRESS_ZLIB |
zlib | Fast | ~60% | Recent chunks |
COMPRESS_LZMA |
lzma | Slow | ~70% | Old chunks |
from amcx import (
AMCXError, # base exception
AMCXInvalidFileError, # not a valid .amcx file
AMCXVersionError, # incompatible version
AMCXCompressionError, # compression/decompression failed
AMCXChunkNotFoundError, # chunk ID not in index
AMCXCorruptError, # CRC32 mismatch
AMCXReadOnlyError, # write attempted on read-only file
AMCXSecurityError, # bypass/jailbreak threat detected
)from amcx import AMCXReader, AMCXCorruptError, AMCXChunkNotFoundError
try:
with AMCXReader("memory.amcx") as reader:
text = reader.read_chunk_text(99)
except AMCXChunkNotFoundError:
print("Chunk 99 does not exist")
except AMCXCorruptError as e:
print(f"Corruption detected: {e}")from amcx import (
AMCXWriter, AMCXReader,
CHUNK_LORE, CHUNK_CHARACTER, CHUNK_ACTIVE,
COMPRESS_LZMA, COMPRESS_ZLIB,
MirrorMode,
)
# Write
writer = AMCXWriter(mirror=MirrorMode.AUTO, recovery=True)
writer.add_text_chunk(0, CHUNK_LORE, "The world", "Dark and vast...", COMPRESS_LZMA)
writer.add_text_chunk(1, CHUNK_CHARACTER, "Aria", "An elven warrior...", COMPRESS_LZMA)
writer.add_text_chunk(2, CHUNK_ACTIVE, "Current session","The group arrives...", COMPRESS_ZLIB)
writer.save("story.amcx")
# Read
with AMCXReader("story.amcx") as reader:
print(reader.summary())
print(reader.read_chunk_text(1)) # Aria's descriptionamcx includes a built-in bypass/jailbreak detection system in detection.py. It scans both the .amcx file contents and the process's RAM for known prompt-injection patterns, and can purge or block execution automatically.
- Prompt injection (
"ignore previous instructions","disregard the system prompt", etc.) - Role confusion and jailbreak framing (
"you are now DAN","act as if you have no rules") - Delimiter smuggling (fake
[system]tags,### system override) - Base64 and Unicode-confusable obfuscation of the patterns above
- Narrative framing (hiding a harmful request inside "imagine a story where...")
- Multi-agent decomposition (splitting a request into innocent-looking fragments meant to be reassembled)
- Long-context manipulation (burying an injected instruction deep inside a wall of text)
- Prompt leak attempts (
"reveal your system prompt","what are your original instructions")
from amcx import scan_chunks
result = scan_chunks("chat.amcx")
if not result.clean:
for threat in result.chunk_threats:
print(threat.chunk_id, threat.matched)from amcx import scan_ram
result = scan_ram(purge=1) # purge=1 zeroes out matched strings in memory
# with the native accelerator
result = scan_ram(purge=1, accelerator_path="./amcx_accel.so")Matched content is never exposed in the result — only "[redacted]" plus which pattern triggered it, to avoid leaking user conversation data even in your own logs.
from amcx import full_scan
result = full_scan(
"chat.amcx",
chunk_scan=1, # 1 = scan file chunks, 0 = skip
ram_scan=1, # 1 = scan process RAM, 0 = skip
ram_purge=1, # 1 = zero out matches found in RAM, 0 = detect only
)Blocks a function from running at all if a threat is detected beforehand:
from amcx import guarded, AMCXSecurityError
@guarded("chat.amcx")
def run_model():
...
try:
run_model()
except AMCXSecurityError as e:
print("execution blocked:", e)Pass raise_on_threat=False to let the function run anyway without raising:
@guarded("chat.amcx", raise_on_threat=False)
def run_model():
...Several parts of amcx have optional C++ accelerator add-ons. They're entirely opt-in — without them, everything runs in pure Python with identical results.
| Add-on | Speeds up | Used by |
|---|---|---|
amcx_accel |
Bypass pattern matching |
scan_chunks, scan_ram, full_scan, guarded
|
amcx_sha1 |
SHA-1 hashing | AMCXMirror.build_block/verify/embed/update |
amcx_xor |
XOR parity & recovery | AMCXRecovery.append/recover_chunk |
amcx_crc32 |
Not currently used — zlib.crc32 is already native C |
— |
result = full_scan("chat.amcx", accelerator_path="./amcx_accel.so")
@guarded("chat.amcx", accelerator_path="./amcx_accel.so")
def run_model():
...If the path doesn't exist, isn't a valid binary, or doesn't expose the expected functions, amcx silently falls back to pure Python — no exception, no broken behavior. See the Add-ons branch for the source and precompiled binaries.
Thank you for your interest in contributing to amcx!
- Fork the repository on GitHub
- Create a branch:
git checkout -b my-feature - Make your changes
- Run tests:
pytest tests/ - Commit:
git commit -m "description of change" - Push:
git push origin my-feature - Open a Pull Request
pip install pytest
pytest tests/ -vAll tests must pass before submitting a PR.
- Python 3.10+
- Type hints on all public functions
- Docstrings on all public methods
- No external dependencies (stdlib only)
Open an issue on GitHub with:
- Python version
- amcx version (
python -c "import amcx; print(amcx.__version__)") - Steps to reproduce
- Expected vs actual behavior
- Encryption support (FLAG_ENCRYPTED is reserved)
- CLI tool (
amcx inspect file.amcx) - Async support for
SmartMemory - More compression algorithms
Wiki made by Claude
1. Why is it called eXtended?
- because there was a base version called Adaptive Memory Chunk (AMC) and this public version has more features
2. When will AMCL be released?
- It will be released at the end of 2026 or the beginning of 2027, although the date may vary.
3. If I use amcl, can I switch to amcx?
- Yes, you can switch using the
.exeexecutable orappimage, and for environments without a UI...
4. Will it have media file support?
- Yes, I plan to add support in the future, although I prefer to finish 'AMCL' first