mem cleaner

mem-cleaner post-processes files your AI coding agent has just written. It runs them through a configurable pipeline whose stages are tools it calls "providers", and it replaces a file only after the result has passed validation. If anything goes wrong it rolls back. It does one job: it strips invisible Unicode carriers (zero-width characters, bidi controls, tag characters) from the prose parts of a file, meaning comments and documentation text, and leaves code, directives and string literals alone.

Status: in development, not yet in the extension catalogue. mem-cleaner is not listed in MEM's extension manifest, so MEM will not offer to install it. The spec and the runner are in the repository for review and early use.

Version	0.1.0 (as declared in its index.md). Not in the catalogue
Commands	MEM CLEAN, with the subcommands SAFE, FULL, TEST, STATUS, PROVIDERS, PIPELINES
Executable content	Yes: runner/*.py, a small standard-library Python program
External actions	No. Providers run as local processes, and remote providers are refused
Requirements	Python 3.8 or newer for the runner. The watermarks-remover provider has to be installed separately (it needs Python 3.10 or newer). The optional full profile also needs a local model reachable on loopback and a configured test_command

Why it exists

Text an agent produces can carry invisible characters such as zero-width spaces, bidi controls and exotic spaces. You can't see them in review, but they still end up in your repository. Removing them sounds trivial until you do it to a whole source file. The tool that does the stripping has no idea what a string literal is, so it removes invisible characters from your test fixtures and regexes just as readily as from your comments.

mem-cleaner exists so this cleanup can happen automatically when you want it, without becoming a new source of bugs. Most of the design is protective machinery:

• it knows which parts of a file are prose and which are code or directives;
• it cleans only the files this agent wrote in this session;
• it validates the result and keeps a journal so it can roll back;
• it only runs unattended if you explicitly approved that.

There is a deliberate idea behind it as well: mem-cleaner itself does not know how to remove anything. It handles capabilities, roles, pipelines, file scope, execution policy and validation. Every actual technique lives behind a provider. Adding a provider means writing a definition file, never changing the engine.

How it works

Pipelines and providers

A pipeline is an ordered list of stages. Each stage has a role (inspect, transform or validate) and asks for a capability, for example unicode. At run time the capability is resolved to whichever provider implements it. Stages always run in this order:

inspect -> transform ... -> rewrite (at most one) -> format -> validate

Two profiles ship:

Profile	What it does	Can run unattended
safe (default)	One deterministic unicode stage over prose regions, then a syntax check	Yes, with approval
full	safe plus one statistical-rewrite stage that rewords prose with a local model	Never

One provider ships: watermarks-remover. mem-cleaner uses two of its operations. "Layer A" removes invisible Unicode characters and normalises exotic spaces to a plain space. A rewrite operation drives a local model through Ollama on 127.0.0.1.

The name safe describes the process (deterministic, allowlisted, region-scoped, validated), not the result. Stripping invisible characters can still change behaviour through a string literal, a regex, a fixture or a snapshot, and all of those still parse fine. That is why validation escalates when a change touches anything other than prose (see below).

Regions: only prose is touched

Before a stage runs, the runner classifies the file into regions:

Region	Examples	Treatment
Prose	Narrative comments, documentation text	May be cleaned
Directives	# type: ignore, # noqa, # pragma, eslint-disable, // @ts-ignore, //go:build	Never touched
Runtime	Docstrings, literals, regexes, fixtures, snapshots	Left alone, or validation escalates to the tests

Only a few file types can be classified today:

File type	Prose is
.py	Narrative # comments only (docstrings count as runtime)
.md, .markdown, .txt	Text outside fenced blocks, inline code and indented blocks
Anything else	Nothing. The file is refused, not treated as all prose

When the runner is unsure, it assumes the riskier region. An unclassifiable region counts as runtime. A Python comment that looks like word: counts as a directive even if the tool it names is unknown. A Markdown line indented four spaces counts as code.

The provider never sees the whole file. The runner extracts only the permitted regions, sends them to the provider in one payload, and splices the result back in. Everything outside those regions is copied byte for byte from the original. Afterwards every byte outside the regions is compared with the original, and any difference rejects the candidate. To see how a file will be split before you trust a pipeline on it, run classify <file> --show.

Which files get cleaned

Running MEM CLEAN without a target covers only files this agent changed in this session. Those come from an attribution record that the agent keeps as it works. After its last write to a file, the agent records the path and the file's hash. If the hash no longer matches when cleaning starts, a human has edited the file since, and it is skipped and reported.

The set is never derived from git diff, because that would include work you did before the session started. If you give an explicit target (MEM CLEAN <target>), that target is used, since it is your instruction. Every skipped file is listed along with the reason, because a file that is silently dropped looks the same as one that was cleaned.

Validation

Some checks run on every candidate before anything is replaced: the content is not empty, it is valid UTF-8, line endings are unchanged, the trailing newline is unchanged, and the file still parses where a parser exists. If there is no parser for a file type, the report says "unchecked" rather than "valid".

Stronger levels are available, and each one needs only its own command:

Level	Adds	Needs
syntax	The checks above	Nothing
format	The project's formatter agrees with the result	extensions_cleaner_format_command
project	The project still builds	build_command
tests	The tests still pass	test_command

The level actually applied is the stronger of what you configured and what the change demands. A change that touches a runtime region demands tests, and so does any rewrite stage. An escalation that cannot be satisfied is a refusal, not a downgrade. If no test command is configured, automatic mode skips the transformation and reports it. An explicit MEM CLEAN states the reduced guarantee and asks you first.

Configured commands run without a shell. A command containing |, &&, > or ; is refused, because without a shell it would quietly do something other than what was written. If you need a pipeline of commands, put them in a script your project owns and configure that script.

Transactions and recovery

No file is modified in place. Each run is journalled (prepared -> installed -> validating -> committed, or rolling-back -> restored on failure), and backups are discarded only after validation passes. Every replace is a compare-and-swap on the original hash, so a concurrent edit or a rebase aborts the swap and is not overwritten. Each file is replaced atomically. A batch of files is recoverable, but not atomic.

If a run is interrupted, every other command refuses until you run recover. Recovery restores what it safely can. It never overwrites a file that was edited after the interrupted run: it reports both hashes and keeps the journal until a human sorts it out.

Installing and first setup

Because mem-cleaner is not in the catalogue, MEM's "which extensions do you want to install?" question will not list it. MEM loads an extension when its files are present under extensions/ in your knowledge base and it is registered active in extensions/EXT.md. So you install this one by hand from the repository's src/extensions/mem-cleaner/ folder. As with any extension that is not in the catalogue, the agent will disclose where it came from the first time it is used in a session.

Setup involves a few things that are specific to this extension:

1. Allow executable content. Every subcommand runs the Python runner, so nothing works while extensions_allow_executable_content is false (the default). In that state the agent must tell you the extension is installed but inactive, and ask whether to enable the setting. It names runner/*.py as the code that would run, and notes that the setting applies to every installed extension that ships code. It changes MEM.config.md only after an explicit yes.

2. Choose the autonomy mode. The agent asks you this at installation and records the answer. That answer is the approval, so no default is proposed:

   Mode	Behaviour
   automatic	Before reporting work complete (after a green test suite), the configured pipeline runs over the files this agent changed
   confirm	Same, but asks yes or no each time
   manual	Nothing runs on its own. Only an explicit MEM CLEAN

3. Install the provider yourself. The agent never installs a provider or the runner. Get watermarks-remover from upstream. The shipped definition expects it under the configured project root (${workspace}). If you keep the checkout somewhere else, pin the path in a custom provider definition rather than editing the shipped one, because an update replaces the shipped file.

4. Check the result with MEM CLEAN STATUS. It shows the interpreter, the runner and Python versions, the resolved project root and where transient state is stored.

Also add __pycache__/ to your ignore rules, because the runner generates bytecode where it runs.

Commands

Command	What it does	Replaces files
MEM CLEAN	Runs the configured pipeline over this session's eligible files	Yes
MEM CLEAN <target>	Runs the configured pipeline over an explicit target	Yes
MEM CLEAN SAFE <target>	Runs the safe profile only (deterministic stages)	Yes
MEM CLEAN FULL <target>	Runs the full profile, including the rewrite stage	Yes
MEM CLEAN TEST <target>	Runs the pipeline transactionally and reports, leaving the target unchanged	No
MEM CLEAN STATUS	Shows configuration, versions, compatibility and state location	No
MEM CLEAN PROVIDERS	Lists providers, their roles and whether they are available	No
MEM CLEAN PIPELINES	Lists pipelines and their stages	No

TEST leaves the target as it was, but a formatter, build or test suite it runs can still write caches or logs. Only STATUS, PROVIDERS and PIPELINES write nothing at all. If a file name could be mistaken for a subcommand, prefix it with ./. For example, MEM CLEAN ./STATUS cleans the file named STATUS.

Every activation starts with an acknowledgement line that names the pipeline and the mode, for example > MEM CLEAN: active (pipeline safe, mode manual). The report then lists each stage, how many changes it made, and what was skipped or refused. It never includes file contents.

Under the hood the agent drives the runner directly: python <KB_ROOT>/extensions/mem-cleaner/runner/cli.py <subcommand>. The runner subcommands are status, providers, pipelines, validate-pipeline, classify, record, recorded, forget, approve, run, test and recover. It finds the knowledge base from its own location, so it works from any directory. --kb-root overrides that.

Configuration

All keys live in MEM.config.md. Unprefixed cleaner_* keys are not read.

Key	Default	Meaning
extensions_cleaner_enabled	true	Turns the extension off when false
extensions_cleaner_mode	"manual"	automatic, confirm or manual. Asked at installation. A missing or unreadable value means manual
extensions_cleaner_pipeline	"safe"	Default pipeline
extensions_cleaner_max_rewrite_stages	1	Pipelines with more rewrite stages are refused. You can raise it, with a warning
extensions_cleaner_validation	"syntax"	Minimum validation level: syntax, format, project or tests
extensions_cleaner_fail_policy	"restore"	What happens when validation fails
extensions_cleaner_include	[]	fnmatch patterns, relative to the project root, that a file must match
extensions_cleaner_exclude	[]	fnmatch patterns to skip
extensions_cleaner_update_daily_log	false	Writes one summary line to the daily log after a run
extensions_cleaner_rewrite_model	null	The local model the rewrite stage uses. There is no default, and without one the stage refuses
extensions_cleaner_format_command	null	Formatter command for format validation
extensions_cleaner_project_root	".."	Where validation commands run, relative to the knowledge base root. STATUS prints the resolved path
extensions_cleaner_validation_timeout	600	Seconds before a validation command is killed

The extension also reads the core settings build_command and test_command, but never writes them. The values auto-detect, none and empty all mean that no such command exists.

The runner parses configuration itself, so it accepts only a restricted YAML subset. Block scalars, anchors, aliases, tags and non-empty flow collections are refused rather than guessed at. If another extension's key falls outside that subset, the runner still recovers its own extensions_cleaner_* keys and reports what it could not read.

Safety and permissions

• Executable-content gate. The extension needs extensions_allow_executable_content: true in MEM.config.md. Installing and updating it each need an approval that names the runner files as executable. MEM doesn't claim to sandbox that code. What it offers instead is that the code lives in your repository, so every update arrives as a diff you can read.

• Checkable runner properties. The runner uses only the standard library, so it installs no dependencies. Providers run as subprocesses and are never imported. The engine contains no provider id.

• Approval with a fingerprint. An automatic approval records exactly what was approved:

  • the pipeline and its stages, including each stage's region scope;
  • each resolved provider and its version;
  • the rewrite limit;
  • the validation level.

  If any of those change, the approval lapses and the mode drops to manual, not to a prompt. A prompt would be seen by the same agent whose output is being cleaned. automatic doesn't prompt even when extensions_require_confirmation: true, because it carries a recorded approval. extensions_enabled: false disables everything.

• Base wins. A custom provider or pipeline may not reuse a shipped id. Such a collision is refused, never resolved. Custom ids are namespaced custom/<id>. This is the opposite of mem-toolbox, and it is deliberate: here a definition names a command that may run unattended.

• Custom providers are bound to what they execute. A script inside the repository is bound by its content hash. A binary outside the repository is bound by its path, version and content hash. A bare interpreter such as python, node or a shell can't be bound, so it needs confirmation on every run and is never eligible for automatic. Commands are argv vectors, never shell strings.

• The rewrite stage is fenced in. statistical-rewrite requires an explicitly named target, always escalates validation to tests, and runs at most once per pipeline. The provider definition pins a loopback address. A definition that names a non-loopback host or passes --allow-remote is refused when it is loaded.

• Provenance is kept. metadata-attribution (author, copyright, licence) and c2pa (Content Credentials) are excluded from every shipped profile and from glob-based automation.

What it never does:

• install the runner, a provider or anything else;
• clean a file mid-task (there is no per-write trigger);
• claim a file was AI-generated based on style;
• report success when validation was skipped;
• treat removal as proof that no watermark remains.

Where its files and data live

What	Where
Spec, commands, shipped providers and pipelines, runner	extensions/mem-cleaner/ in your knowledge base (base layer, replaced by updates)
Your own providers and pipelines	extensions/mem-cleaner/custom/providers/ and custom/pipelines/ (never touched by an update)
Per-machine data (resolved Python interpreter)	extensions/mem-cleaner/custom/host/<host>.md, created at installation
Journals, backups, attribution records, approval	The platform state directory: %LOCALAPPDATA%\mem-cleaner\ on Windows, $XDG_STATE_HOME/mem-cleaner/ (or ~/.local/state/mem-cleaner/) elsewhere. Keyed per knowledge base, never committed

The attribution record holds only paths and hashes, never content. After a successful run the record is re-baselined, so the extension's own edit isn't mistaken for a human one later.

Limits and known issues

• Not in the catalogue. You have to install it by hand, and update checks have no catalogue version to compare it against.
• Narrow coverage. Only .py, .md, .markdown and .txt files can be classified. Every other file type is refused by scoped stages.
• One provider. The metadata-technical stripping offered upstream is deliberately not used, because upstream's results depend on which optional system tools happen to be installed. There is no balanced profile.
• full is a deliberate, costly act. Rewording changes tone and precision and isn't reproducible. The pinned low temperature (0.3) favours a faithful file over a thorough scrub. It needs a local model, extensions_cleaner_rewrite_model and test_command, and refuses if any of them is missing.
• Removal is not proof of absence. Layer A does not remove token-sampling watermarks. Soft bindings and trigger-phrase marks are out of scope.
• Batches are recoverable, not atomic. A long tests validation can be slow, and validation commands are killed after the configured timeout.
• Docs and runner are not fully in sync. Parts of the in-repo documentation still describe an earlier state, for example "no rewrite provider exists", or a metadata-technical stage in example output. Where they disagree, trust the shipped pipeline and provider definitions.

See also

• Spec: mem-cleaner/index.md
• Extensions: how MEM extensions, the manifest and executable content work
• mem-toolbox: the tool registry. It can attest that a provider's executable is present, but it never runs or changes anything