Repository navigation
v0.2.0
Added
- HEIC, AVIF, JPEG XL and ICO files are recognized, checked and accepted by default
(FileType::Heif,Avif,Jxl,Ico). Narrow the set withPolicy::with_allowed_types
or--typesif your service does not take them. - BigTIFF is parsed; it used to be reported as
image.malformed. - PSD layer records are walked: a broken record is
image.malformed, and a layer larger
than the pixel limit isimage.pixel_bomb. - PDFs encrypted with an empty user password are decrypted and inspected.
pdf.encrypted
is then reported at Info instead of Medium.ScannerBuilder::pdf_passwordsand
--pdf-passwordssupply passwords to try on the others, as user or owner password. pdf.trailing_dataandpdf.polyglotfor bytes after the last%%EOF. A PDF with a ZIP
archive appended used to come back clean.- XFA forms are searched for scripts, reported as
pdf.javascript. - Files embedded in a PDF are matched against the YARA rules.
- Brotli-compressed metadata boxes in JPEG XL are decompressed and searched
(image.metadata.payload). - The
cdrfeature:Scanner::sanitizeandmalmal::cdr::rebuilddecode a clean raster
image and write a new file from its pixels.malmal sanitize INPUT OUTPUTand the
daemon'sPOST /sanitizedo the same;malmal-clienables the feature by default.
Animations keep their frames, delays and loop count; RGB, grey and CMYK colour profiles
are applied, so that the new file, which has none, shows the same colours.malmal sanitizeputs the
new file in place of the old one in a single step, or leaves the old one alone. - Rebuilding beyond raster images: JPEG XL (as PNG); SVG with the
cdr-svgfeature,
filtered through an allowlist and checked again by the SVG analyzer; PDF with the
cdr-pdffeature, as pictures of its pages, at--pdf-dpi, optionally as JPEG
(--pdf-jpeg), and opened with the passwords supplied for scanning.malmal-cliand
malmal-ffienablecdr-svgby default andcdr-pdfon request.
cdr::Options::typesnames the types that are rebuilt. Scanner::sanitizescans the file it has rebuilt and refuses it when that scan is not
clean (cdr::Error::Unclean). Found by fuzzing: pixels that spell a PDF, hidden by the
compression of the upload, came out of the PNG encoder as they were, and the rebuilt
file was an image and a PDF with a script at once.- Deeper PDF checks, for what a keyword does not show. New findings:
pdf.dangerous_uri(links that run a script, start a Windows protocol handler, or open
a local file or network share),pdf.external_ref(streams, forms and file
specifications whose content is outside the file),pdf.xml_entityand
pdf.xml_external_ref(DOCTYPE, entities and XInclude in metadata and XFA),
pdf.pixel_bomb,pdf.page_size,pdf.draw_recursion(forms that draw themselves),
pdf.conflicting_objects(one object defined twice in one revision), and at Info
pdf.risky_codec,pdf.print_onlyandpdf.signature_modified.pdf.postscriptnow
covers thePSoperator in a form's appearance string and calculator functions that
hold more than a calculator's operators;pdf.rich_mediacoversMovie,Soundand
Screenannotations andU3DandPRCartwork;pdf.polyglotcovers ZIP headers and
HTML or SVG markup outside every stream;pdf.malformedcovers a second header,
decode parameters outside the format's ranges and names over 255 bytes. malmal scanandmalmal sanitizemap the file instead of reading it: a scan of a
large file needs the memory the analysis keeps, not the size of the file, and a 1 GB
PDF is scanned in a container limited to 96 MB. Files of 4 MiB and more are hashed and
searched for payloads on threads of their own while the format is analyzed, and what is
looked for outside the streams of a PDF is no longer looked for inside them.--no-hashandScannerBuilder::without_hash: no SHA-256, which is most of the time
of a large file on a processor without SHA instructions.Report::sha256is then
empty. A blocklist brings the hash back. 1 GB: 8.0 s before, 4.4 s now, 0.42 s without
the hash.- The values parsed from a PDF take about half the memory they did, and the number kept
grows with the size of the file (one for every 8 bytes, 2 million at least), so that
a large document of many objects is less likely to stop atlimit.complexity. - The
evaluateexample measures refusal and false-refusal rates on files whose nature
is known, with the findings that decided each. On CIC-Evasive-PDFMal2022 (9,107 clean
and 21,899 malicious files) 98.19 % of the malicious files are refused and 87.32 % of
the clean ones accepted; every clean file refused carries what it is refused for
(scripts, XFA forms, attachments, launch actions). See the README. - A PDF is no longer kept in memory as a tree of its values. An object is parsed where
it stands each time a check looks at it; only the values that references lead to are
kept. 100,000 small objects: 87 MB before, 22 MB now, and no slower. A file of 500,000
objects, which stopped atlimit.complexity, is analyzed to its end in 94 MB. - Content streams are read a block at a time as they are inflated, through every filter
of the chain. A page stream of 200 MB once inflated: 256 MB before, 12 MB now.
Object streams, attachments, XFA packets and metadata are still decoded whole, within
Limits::max_decompressed. malmald --spool-dir <DIR>: files of 1 MiB and more are written to unnamed files in
DIRwhile they are scanned and mapped from there, so that the memory of the daemon
does not grow with the files it is sent. The files are opened before the sandbox
closes, one for each worker, and used again: the sandbox allows nothing more than it
did. A 1 GB file is scanned over each of the four endpoints by a daemon limited to
96 MB. Without the option every file is held in memory, as before.--max-decompressedand--max-pixelsfor both commands, which with--max-sizeand
--workersdecide what a request can cost in memory.--allow <CODES>for both commands: the finding codes named, separated by commas or
|, are reported at Info and leave a file clean, asPolicy::with_overridedoes from
Rust. An unknown code is an error, and so is alimit.*code.malmal::report::CODES
lists every code the scanner reports, andPolicy::allowchecks a code against it
(AllowError).Scanner(allow=[...])in Python andmalmal_scanner_new_with_allowin
C take the same list; the C function also gives the reason when it returns NULL. The README lists the codes honest files carry and
what letting each through takes on.malmal_sanitizein the C interface andScanner.sanitizein the Python binding;
malmal_scanner_new_with_passwordsandScanner(pdf_passwords=...)for encrypted PDFs.malmald --connections. Every connection has its own thread, and--workersnow counts
the requests worked on at once across all endpoints, not the connections on each: a
client that keeps its connection open no longer holds a worker while idle.--hard-timeout: a scan still running after it ends the process, where the scan's own
time limit, checked between steps, could be waited for in vain.malmald --icap: ICAPREQMODandRESPMODfor proxies. Multipart forms are taken
apart and each uploaded file is scanned.- Release archives with static binaries for x86-64 and ARM64, a container image, a Python
binding inbindings/pythonwith wheels for Linux, and metadata forcargo binstall. malmald --http:POST /scan,GET /metrics(Prometheus),GET /healthz.malmald --clamd: theINSTREAMcommand of the ClamAV daemon protocol.--max-sizeand--scan-timeoutfor both commands;Scanner::limits().docs/findings.mdanddocs/integration.md; thesurveyexample.
Changed
FileType,ReportandFindingare#[non_exhaustive].malmal::payload,BudgetandLimitExceededare no longer public. They were internals
that no caller needs.malmald --socketis no longer required on its own: one of--socket,--http,
--clamdor--icapis.- New dependencies:
md-5andaes, for PDF decryption;brotli-decompressor, for JPEG XL
metadata;image,png,moxcms,zune-jpeg,zune-coreandjxl-oxide, only with
thecdrfeature;svg-hushwithcdr-svg;hayrowithcdr-pdf;memmap2in
malmal-cli.
Fixed
pdf.javascriptandpdf.embedded_filewere reported for name trees that list
nothing,pdf.embedded_filefor the packets of an XFA form and for the rights of a
signature (/EFas an array). 428 of the 9,107 clean files of CIC-Evasive-PDFMal2022
were refused for one of these.sanitizeheld the pixels of an image twice while writing it: a PNG at the limits
took 494 MB and takes 250 MB.- A PDF of thousands of streams that all run to one
endstreamwas read once for each
of them: 8 s for a file of 800 KB. The work is now cut off withlimit.complexity,
and the time limit is checked between streams.
Behaviour to check when upgrading
-
A link to a file on the reader's own machine that is not a program (
file:///C|/…)
is no longerpdf.dangerous_uri; a network share or a program still is. -
limit.complexitycounts objects and the values that references lead to, one for
every 16 bytes of the file and two million at least, and no longer every value in the
file. Documents of many objects that were refused are now analyzed. -
The address-space limit of the sandboxed daemon grows with
--workers,--max-size
and--max-decompressedbeyond their defaults. -
malmalandmalmaldexit with status 3 when a scan runs for six times the scan
timeout. Run the daemon under a supervisor that restarts it, or pass--hard-timeout 0. -
PDFs that came back clean may now be suspicious or malicious for what the deeper
checks find. Run thesurveyorevaluateexample on your own files before deploying,
and lower a finding withPolicy::with_overrideif your users need what it reports. -
The address-space limit of the sandbox grows with
--max-sizebeyond its default. -
Files that were
type.unknownbecause they were HEIC, AVIF, JPEG XL or ICO are now
analyzed and may be clean. -
PDFs with data after
%%EOFare now at least suspicious. -
Encrypted PDFs that open without a password may now be clean, or malicious for what was
found inside.