Repository navigation
Releases
v0.5.1
Fixed
-
A JPEG XL whose JPEG reconstruction data names a component that is not there, or a
scan with an impossible spectral selection or approximation, is rebuilt from its
pixels; the decoder used to panic on it (malmal-jxl-jbr0.2.2). -
A duotone PSD is shown in the colours Photoshop shows it in, which it writes in the
file for other programs (resource 1066); an ink of a colour book such as Pantone,
whose colour is not in the file otherwise, used to print black. An overprint colour
the file leaves unset is the product of its inks, as for a duotone of two, rather
than black. -
A TIFF of JPEG pages with a page of another compression keeps the coefficients of
its JPEG pages; one such page used to send the whole file through its pixels. -
Lossless WebP is written about twice as fast and some 2 to 3 percent smaller: far
copies of a few pixels, which cost more than the pixels, are no longer made.
Added
Hdr::Sdr(--tone-map,tone_mapin the bindings) tone maps an AVIF of high
dynamic range (PQ, HLG) into sRGB, for viewers that do not read thecICPchunk of
the 16-bit PNG it is kept as otherwise.- Old-style JPEG in TIFF (compression 6) in tiles, and of more than one page: the
pages become a TIFF of JPEG pages of the later kind, their coefficients kept. - A weekly
Forksworkflow checks the crates copied intothird_party/for RustSec
advisories under their upstream names, and warns once upstream releases a newer
version.
v0.5.0
Changed
- A rebuilt JPEG keeps the quantised coefficients of the original, with Huffman tables
of its own: its pixels do not change and it is about the size it was. Exif rotations
are done on the coefficients where the image is whole blocks. Where a JPEG has to be
encoded again (a colour profile or inks to convert, a rotation that cuts through
blocks) it is encoded at 4:4:4 with no coarser a quantisation than it had. - An ICO is rebuilt as an ICO with every image, not as a PNG of the largest.
- A PNG of 256 colours or fewer is written indexed when that is smaller, and rebuilt
images keep their pixel density (PNG, BMP, TIFF, PSD, JPEG). - TIFF: palette, CIE Lab* (with its white point), planar, CMYK through its profile,
and associated alpha are read as they look. - A lossy WebP stays lossy: every symbol of its VP8 stream is coded again at the same
probabilities, so it decodes to the same pixels and is about the size it was, where it
used to come out as a lossless WebP many times larger. Its alpha is coded again
without loss, and animations keep every frame as it was placed and blended. The file
is kept only if it decodes to exactly the pixels the original does. - A GIF keeps its palettes and the indices of its pixels, frame by frame, instead of
being quantised again from full colour; an animated PNG keeps its frames as they were
laid out, at 16 bits when it has them. - A TIFF of several pages is rebuilt as a TIFF of all of them (Deflate), not as a PNG
of the first. CCITT fax (Modified Huffman, T.4 one- and two-dimensional), LZMA,
Zstandard and WebP compression are read, and samples of any depth from 1 to 64 bits,
integer or floating point. - An animated JPEG XL is rebuilt as an animated PNG of every frame, with its delays.
- PSD: composites compressed with ZIP, with or without prediction, and multichannel
documents, drawn in the colours of their inks; duotones in the colours of their inks,
through their curves. - A JPEG XL made from a JPEG gives back the JPEG, rebuilt with its coefficients kept,
where it shows what the JPEG XL does. - A TIFF whose strips or tiles are JPEG pictures is rebuilt as one JPEG of their
coefficients, without decoding them. - A lossy WebP turned by its Exif stays lossy, with Exif of nothing but its orientation.
- A TIFF of one page of floating-point samples stays a TIFF of them, past black and
white too, instead of a PNG clipped to 16 bits. - Lossless WebP is written by an encoder of malmal's own (colour table, prediction block
by block, the colour transform, a colour cache, codes for groups of blocks, copies of
earlier pixels), about 5 to 60% smaller; the alpha of a lossy WebP is filtered the way
that codes smallest. Each is decoded again and kept only if every pixel comes back. - AVIF is decoded on up to four threads for large images, and stops at
max_duration
instead of running on unwatched. - A TIFF of several pages of JPEG pictures stays a TIFF of them, each page its
coefficients put together in one strip; with--keep-cmykso does one of inks.
Every page is checked against what its pixels show. - JPEG in a TIFF of the old kind (compression 6), whole or as tables and a scan, is
rebuilt as the JPEG it stands for. - A TIFF of grey floating-point samples stays grey.
- A duotone's overprint colours are used where its inks print over one another.
- AVIF for high dynamic range displays (PQ, HLG) is rebuilt as a PNG of 16 bits a
channel that names its primaries and transfer (cICP), instead of being refused.
Added
-
PSD and PSB files are rebuilt as a PNG of their composite image, with transparency
where Photoshop says there is any. -
AVIF files are rebuilt as a PNG of their primary image, decoded by
filmcraft-av1
(safe Rust, the same samples as libdav1d): grids of tiles, an alpha image, clean
aperture, rotation and mirroring, and colours brought into sRGB from an ICC profile or
the primaries and transfer the file names. Images for high dynamic range displays
are refused. -
Options::inks(Inks::Keep),malmal sanitize --keep-cmykandkeep_cmykin the C,
Python and Ruby interfaces: a CMYK JPEG, TIFF or PSD keeps its inks, a JPEG its
coefficients and a TIFF or PSD its samples (written as a TIFF), with its colour
profile rebuilt from the tags that define its colours, copied byte for byte and
checked to print every ink as the original does. -
Options::profiles(Profiles::Keep),malmal sanitize --keep-profileand
keep_profilein the C, Python and Ruby interfaces: colours under an RGB profile
other than sRGB (Display P3, Adobe RGB) stay as they are in a JPEG (its coefficients),
PNG, TIFF, WebP (lossy too), BMP, GIF, animated PNG, JPEG XL, PSD or AVIF, with the
profile rebuilt from the tags that define its colours and checked on a 17³ grid at 16
bits; shades under a grey profile likewise, checked at 257 levels. An AVIF that names
its primaries and transfer keeps them in acICPchunk. -
third_party/: jxl-oxide with the fixes to JPEG reconstruction of its open pull
requests (tirr-c/jxl-oxide#508, #510, #511) and of issues #507 and #512, and
filmcraft-av1 with a flag that stops decoding, published asmalmal-*crates until
upstream releases them. A progressive JPEG with subsampled chroma made into a JPEG XL
now gives back its JPEG instead of panicking.
v0.4.2
Added
malmald --https,--icapsand--clamd-tls: HTTP, ICAP and the ClamAV protocol over
TLS, with--tls-certand--tls-key(PEM).--tls-client-camakes their clients
present a certificate signed by a CA in the file, which authenticates the ClamAV
protocol from other hosts. TLS is thetlsfeature ofmalmal-cli, on by default;
building it for musl needs a musl C compiler (musl-gcc) forring.malmald --token-filecovers ICAP too:REQMODandRESPMODhave to carry
Authorization: Bearer <token>(Squid:adaptation_meta), or are answered401
before their body is read.OPTIONSneeds no token.- YARA rules can import the
peandelfmodules.
Changed
- The
yarafeature builds for musl, into a static binary, given a musl C compiler
(CC_x86_64_unknown_linux_musl=musl-gcc); CI builds it and runs it on a bare Debian. - The start-up warning now names ICAP endpoints without a token, and leaves out TLS
endpoints that ask for client certificates.
v0.4.1
Fixed
- Ruby: an interrupt during a scan or a rebuild, such as
Timeout.timeoutor
Thread#raise, no longer loses the report and the rebuilt file the library allocated;
it is raised once they are freed.Malmal::Scanner#closewhile another thread scans
no longer frees the scanner under it: the scanner is freed as the last scan running
returns, and scans aftercloseraiseMalmal::ScanError.
v0.4.0
Added
-
malmal_scanner_new_jsonin the C interface: one JSON object carries every option
(policy,allow,types,hashlist,pdf_passwords, the limits,max_memory,
without_hash, andwait, which sendsmalmal_scanthroughScanner::scan_within).
An unknown key is an error. Alsomalmal_scanner_overdue,malmal_versionand
malmal_sanitize_json, which takes the options of the rebuild (types,pdf_dpi,
pdf_jpeg,pdf_text,pdf_vector_text). The older constructors stay. -
Python:
Scanner(policy=, types=, limits=, without_hash=, wait=),Scanner.overdue,
Scanner.library_version, and the rebuild options as keywords ofsanitize. -
Ruby: the
malmalgem, on theffigem overlibmalmal_ffi, which a scan runs with
the GVL released.Malmal::Scannertakes every option of the JSON constructor and
returnsMalmal::Report;#sanitizereturnsMalmal::Sanitizedor raises
Malmal::RejectedorMalmal::NotRebuilt.Malmal::Clientdoes the same with a
runningmalmald, over HTTP with its token. Platform gems carry the library for
x86_64-linux-gnuandaarch64-linux-gnu(glibc 2.17),x86_64-linux-musland
aarch64-linux-musl,arm64-darwinandx86_64-darwin; therubygem uses the library
atMALMAL_LIBRARY. Ruby 3.2 to 4.0, with RBS signatures. -
Rails: the
malmal-railsgem, for Rails 7.1 to 8.1.config.malmalbuilds one scanner
for the application, or a client ofmalmald.validates :file, malmal: truescans
Active Storage attachments before they are stored, signed ids of direct uploads too, and
adds:malmal_suspicious,:malmal_maliciousor:malmal_errorwith the finding codes.
Every stored blob is scanned when it is analyzed, the verdict kept in
blob.metadata["malmal"], and purged if it is not clean with
config.malmal.purge_unclean.sanitizes_attachmentstores a rebuilt file in place of
the upload. Scans and rebuilds arescan.malmalandsanitize.malmalevents;bin/rails generate malmal:installwrites the initializer.examples/railsis an application that
uses it, in the process or besidemalmald. -
The release workflow builds the gems, tests the installed ones on Linux (glibc and musl)
and macOS, attaches them to the release, and pushes them to RubyGems with trusted
publishing when theRUBYGEMS_PUBLISHvariable istrue. -
Policy::documents()and--policy documents: the default policy with XFA forms and
attachments allowed, for services that take documents from people. What they contain
still counts. On CIC-Evasive-PDFMal2022 it accepts 92.38% of the clean files, against
87.32%, and refuses 99.06% of the malicious ones.Policy::namedlooks a policy up by
name. -
malmald --token-file <FILE>: HTTP requests other thanGET /healthzhave to carry
Authorization: Bearer <token>, or are answered401before their body is read. The
daemon also warns at start-up about each endpoint that is not on loopback and that
anyone can use. -
Scanner::scan_within: a scan that answers within a given time, even when a step of it
does not end. The scan runs on a thread of its own; when it is late, the report has
limit.timeand the scan is left running, at mostMAX_OVERDUE(4) of them, past which
files are refused at once.Scanner::overduecounts them. -
--max-memory(512 MiB) formalmalandmalmald, andScannerBuilder::memory_limit
withMemoryLimitin the library: a scan that holds more than that beyond what its
thread held when it began is stopped withlimit.memory. The tools count what each
thread allocates; the limit is checked as each block of a PDF stream is decoded and
between steps. Two files that inflate to 200 MiB peaked at 205 MiB without it and
55 MiB with a limit of 64 MiB. -
pdf.c2pa_manifest(Info): attachments that are all C2PA manifests (Content
Credentials), checked down to their JUMBF boxes, are no longerpdf.embedded_file. What
the manifests carry is still scanned. -
pdf.reference_loop(Medium): a page tree, outline, chain of actions or chain of object
streams that leads back to itself. -
pdf.link_lure(Low): a link to a web page that covers a fifth or more of a page. On
CIC-Evasive-PDFMal2022 the default policy now refuses 98.65% of the malicious files,
against 98.19%, and accepts as many clean ones, 87.32%. -
image.vendor_trailerrecognizes the trailer that OPPO cameras write after a JPEG
image. -
malmal_scanner_set_max_memoryin the C interface andScanner(max_memory=…)in
Python, 512 MiB by default: the library counts what each thread allocates, as the
commands do. PNG chunks, SVGZ and JPEG XL boxes are inflated in blocks against the
limit, and a PostScript ASCII85 string is held to what is left of it. -
pdf.signature_invalid(Medium): a signature whose byte ranges do not fit the file or
leave more than its value out, that has no value of its own, lies outside what it
signs, or sits in an object stream (signature forgery and wrapping).
pdf.unsigned_content(Medium): page content written after what the last signature
signs. Usage rights signatures are not judged. -
pdf.ambiguous_structure(Medium): structures that readers take for different
documents (content masking): stream lengths and keywords, keys given twice,
cross-reference entries that point elsewhere or at a comment, objects defined again
after the laststartxref. On CIC-Evasive-PDFMal2022, 1,558 malicious files and no
clean one. -
pdf.disguised_link(Medium): a web link through a URL shortener, to a numeric address
or a punycode host, or to a name of made-up syllables. The default policy now refuses
99.07% of the CIC malicious files, against 98.67%, and accepts as many clean ones. -
A rebuilt PDF keeps the text of its pages, where it was, as characters that can be
searched and copied and do not draw;--pdf-no-textandOptions::pdf_textleave it
out. -
A rebuilt PDF draws text from the outlines of its glyphs, written anew as Type 3 fonts,
where the page shows it in one opaque colour on a plain background with nothing over
it; each glyph is checked against the drawing of the page before it is taken out of
it.--pdf-raster-textandOptions::pdf_vector_textkeep all text in the drawing.
On 78 clean CIC-Evasive-PDFMal2022 documents, the 47 with text came out a fifth of
the size they did (median; at most 0.78), with a median 2.5% of the dark pixels of the
page left in the drawing; poppler draws none of them further from the original than
the drawing alone. The other 31 are scans and come out as before. Rebuilding took 51%
longer.
Changed
- The Python package needs a
libmalmal_ffiof this version: it uses the JSON
constructor. PDF passwords may now contain line breaks. - The crates declare
rust-version = "1.95", the oldest Rust that builds them with every
feature (yara-xneeds it); CI builds with it. - The features
zenavif,avif-decode,svg-hushandhayroofmalmalare gone. The
first two did nothing; the others only pulled in a dependency thatcdr-svgand
cdr-pdfuse. - The GitHub Actions the workflows use are pinned to commits.
Fixed
- Page content given as an array that sits in an object stream was not scanned.
- Page content that is not a stream, calculator functions that ask for more values than
their stack holds or compute numbers past floating point on their domain, and filters
the format does not define arepdf.malformed.
v0.3.0
Added
malmald --file-budget: what the files of requests held at once may come to, as they
arrive and while they are scanned. A file takes its share as it arrives and waits for
room; one that finds none within--timeoutis refused. The default,--workerstimes
--max-size, is what the files could take before.malmald --min-rate(1024 bytes a second): once--timeouthas passed, the pace a
request and its answer have to keep, counting only the time spent waiting on the client.
Changed
malmal-clidepends onmemchr, whichmalmalalready used.- The crate documentation says what is checked, how verdicts and the policy work, the
limits, threads, stability and the features, with an example of a configured scanner. - A daemon request takes its worker once its file has arrived, and gives it back before
its answer is written.--workersnow counts scans, not requests.
Fixed
- Files made of one structure repeated many times were read again for each repetition,
and some ran past the time limit, which none of them reached a check of. Measured
before and after on a file of 1 MB: trailers whose strings never close, more than
60 s and 0.02 s; objects that cannot be parsed, 9.1 s and 0.02 s; an object stream
whose entries all point at one value, more than 60 s for 40 KB and 0.04 s; forms
that draw a name listed in one large table, 7.1 s and 574 MiB, 0.04 s and 14 MiB;
/Encryptin every object, 54 s for 16 MB and 0.43 s; in SVG,&without;,
34 s and 0.01 s, andurl(without), 18 s and 0.03 s; in PostScript,
%%BeginResourcecomments that one keyword closes, 19 s and 0.04 s, and(that
never closes, the 10 s of the time limit and 0.05 s: one pass tells which strings
close, so ASCII85 image data, which is full of parentheses, is read once; an icon
whose entries all name one image, 34 s for 1.3 MB and 0.01 s. Tags of server-side
code in a raster image took 6.5 s for 16 MB and take 0.17 s. Of these, the trailers
and the object stream now stop atlimit.complexity, and so do PostScript strings
that start after a backslash and never close, which that pass cannot tell of; an icon
whose images lie over one another isimage.malformed. - What a damaged PDF filter produced before it gave out was not counted against
--max-decompressed: 16 streams that inflate to 64 MiB each and end in a row the
predictor rejects, 2.4 MB in all, were clean after 3 s, and 38 MB of them ran to the
time limit. Both now stop atlimit.decompressionin 0.7 s. ASCII85 strings in
PostScript, whosezis four bytes, are counted too. - Four clients sending a byte every five seconds held the four workers of a daemon for as
long as they went on: another request went unanswered for over a minute, and the four
were never let go. The other request is now answered at once, and the four are let go
after--timeout. - A multipart boundary of 60,000 bytes took 7 s of a worker for a body of 4 MiB, outside
the time limit, and takes 0.02 s. The files of one form share the time limit of one
scan; each had its own. - Bodies sent to a
--spool-dira byte at a time cost a write each: 8 MiB took 14 s over
ICAP and 12 s over the ClamAV protocol, and take 2.7 s and 0.6 s, as without a spool.
Behaviour to check when upgrading
- A PDF whose trailers take more than eight times the file to read, an object stream
whose entries do, and a PostScript program whose strings that start after a backslash
and never close take more than four times the program, arelimit.complexity. An
icon whose images take more than twice the file isimage.malformed. None of the files oftests/corpuscomes near these. - A PDF whose damaged streams decode to more than
--max-decompressedin all is
limit.decompression, and so is a PostScript program whose ASCII85 strings do. malmaldmay hold more files at once than it has workers, up to--file-budget: with
the defaults and no spool, 32 uploads of 16 MiB at once took it to 316 MB where 185 MB
before. A request that finds no room within--timeoutis refused (503over HTTP).
A client slower than--min-ratepast--timeoutloses its connection. The spool
opens a file for each MiB of the budget, up to--connections, where one per worker.
v0.2.0
Added
- HEIC, AVIF, JPEG XL and ICO files are recognized, checked and accepted by default
(FileType::Heif,Avif,Jxl,Ico). Narrow the set withPolicy::with_allowed_types
or--typesif your service does not take them. - BigTIFF is parsed; it used to be reported as
image.malformed. - PSD layer records are walked: a broken record is
image.malformed, and a layer larger
than the pixel limit isimage.pixel_bomb. - PDFs encrypted with an empty user password are decrypted and inspected.
pdf.encrypted
is then reported at Info instead of Medium.ScannerBuilder::pdf_passwordsand
--pdf-passwordssupply passwords to try on the others, as user or owner password. pdf.trailing_dataandpdf.polyglotfor bytes after the last%%EOF. A PDF with a ZIP
archive appended used to come back clean.- XFA forms are searched for scripts, reported as
pdf.javascript. - Files embedded in a PDF are matched against the YARA rules.
- Brotli-compressed metadata boxes in JPEG XL are decompressed and searched
(image.metadata.payload). - The
cdrfeature:Scanner::sanitizeandmalmal::cdr::rebuilddecode a clean raster
image and write a new file from its pixels.malmal sanitize INPUT OUTPUTand the
daemon'sPOST /sanitizedo the same;malmal-clienables the feature by default.
Animations keep their frames, delays and loop count; RGB, grey and CMYK colour profiles
are applied, so that the new file, which has none, shows the same colours.malmal sanitizeputs the
new file in place of the old one in a single step, or leaves the old one alone. - Rebuilding beyond raster images: JPEG XL (as PNG); SVG with the
cdr-svgfeature,
filtered through an allowlist and checked again by the SVG analyzer; PDF with the
cdr-pdffeature, as pictures of its pages, at--pdf-dpi, optionally as JPEG
(--pdf-jpeg), and opened with the passwords supplied for scanning.malmal-cliand
malmal-ffienablecdr-svgby default andcdr-pdfon request.
cdr::Options::typesnames the types that are rebuilt. Scanner::sanitizescans the file it has rebuilt and refuses it when that scan is not
clean (cdr::Error::Unclean). Found by fuzzing: pixels that spell a PDF, hidden by the
compression of the upload, came out of the PNG encoder as they were, and the rebuilt
file was an image and a PDF with a script at once.- Deeper PDF checks, for what a keyword does not show. New findings:
pdf.dangerous_uri(links that run a script, start a Windows protocol handler, or open
a local file or network share),pdf.external_ref(streams, forms and file
specifications whose content is outside the file),pdf.xml_entityand
pdf.xml_external_ref(DOCTYPE, entities and XInclude in metadata and XFA),
pdf.pixel_bomb,pdf.page_size,pdf.draw_recursion(forms that draw themselves),
pdf.conflicting_objects(one object defined twice in one revision), and at Info
pdf.risky_codec,pdf.print_onlyandpdf.signature_modified.pdf.postscriptnow
covers thePSoperator in a form's appearance string and calculator functions that
hold more than a calculator's operators;pdf.rich_mediacoversMovie,Soundand
Screenannotations andU3DandPRCartwork;pdf.polyglotcovers ZIP headers and
HTML or SVG markup outside every stream;pdf.malformedcovers a second header,
decode parameters outside the format's ranges and names over 255 bytes. malmal scanandmalmal sanitizemap the file instead of reading it: a scan of a
large file needs the memory the analysis keeps, not the size of the file, and a 1 GB
PDF is scanned in a container limited to 96 MB. Files of 4 MiB and more are hashed and
searched for payloads on threads of their own while the format is analyzed, and what is
looked for outside the streams of a PDF is no longer looked for inside them.--no-hashandScannerBuilder::without_hash: no SHA-256, which is most of the time
of a large file on a processor without SHA instructions.Report::sha256is then
empty. A blocklist brings the hash back. 1 GB: 8.0 s before, 4.4 s now, 0.42 s without
the hash.- The values parsed from a PDF take about half the memory they did, and the number kept
grows with the size of the file (one for every 8 bytes, 2 million at least), so that
a large document of many objects is less likely to stop atlimit.complexity. - The
evaluateexample measures refusal and false-refusal rates on files whose nature
is known, with the findings that decided each. On CIC-Evasive-PDFMal2022 (9,107 clean
and 21,899 malicious files) 98.19 % of the malicious files are refused and 87.32 % of
the clean ones accepted; every clean file refused carries what it is refused for
(scripts, XFA forms, attachments, launch actions). See the README. - A PDF is no longer kept in memory as a tree of its values. An object is parsed where
it stands each time a check looks at it; only the values that references lead to are
kept. 100,000 small objects: 87 MB before, 22 MB now, and no slower. A file of 500,000
objects, which stopped atlimit.complexity, is analyzed to its end in 94 MB. - Content streams are read a block at a time as they are inflated, through every filter
of the chain. A page stream of 200 MB once inflated: 256 MB before, 12 MB now.
Object streams, attachments, XFA packets and metadata are still decoded whole, within
Limits::max_decompressed. malmald --spool-dir <DIR>: files of 1 MiB and more are written to unnamed files in
DIRwhile they are scanned and mapped from there, so that the memory of the daemon
does not grow with the files it is sent. The files are opened before the sandbox
closes, one for each worker, and used again: the sandbox allows nothing more than it
did. A 1 GB file is scanned over each of the four endpoints by a daemon limited to
96 MB. Without the option every file is held in memory, as before.--max-decompressedand--max-pixelsfor both commands, which with--max-sizeand
--workersdecide what a request can cost in memory.--allow <CODES>for both commands: the finding codes named, separated by commas or
|, are reported at Info and leave a file clean, asPolicy::with_overridedoes from
Rust. An unknown code is an error, and so is alimit.*code.malmal::report::CODES
lists every code the scanner reports, andPolicy::allowchecks a code against it
(AllowError).Scanner(allow=[...])in Python andmalmal_scanner_new_with_allowin
C take the same list; the C function also gives the reason when it returns NULL. The README lists the codes honest files carry and
what letting each through takes on.malmal_sanitizein the C interface andScanner.sanitizein the Python binding;
malmal_scanner_new_with_passwordsandScanner(pdf_passwords=...)for encrypted PDFs.malmald --connections. Every connection has its own thread, and--workersnow counts
the requests worked on at once across all endpoints, not the connections on each: a
client that keeps its connection open no longer holds a worker while idle.--hard-timeout: a scan still running after it ends the process, where the scan's own
time limit, checked between steps, could be waited for in vain.malmald --icap: ICAPREQMODandRESPMODfor proxies. Multipart forms are taken
apart and each uploaded file is scanned.- Release archives with static binaries for x86-64 and ARM64, a container image, a Python
binding inbindings/pythonwith wheels for Linux, and metadata forcargo binstall. malmald --http:POST /scan,GET /metrics(Prometheus),GET /healthz.malmald --clamd: theINSTREAMcommand of the ClamAV daemon protocol.--max-sizeand--scan-timeoutfor both commands;Scanner::limits().docs/findings.mdanddocs/integration.md; thesurveyexample.
Changed
FileType,ReportandFindingare#[non_exhaustive].malmal::payload,BudgetandLimitExceededare no longer public. They were internals
that no caller needs.malmald --socketis no longer required on its own: one of--socket,--http,
--clamdor--icapis.- New dependencies:
md-5andaes, for PDF decryption;brotli-decompressor, for JPEG XL
metadata;image,png,moxcms,zune-jpeg,zune-coreandjxl-oxide, only with
thecdrfeature;svg-hushwithcdr-svg;hayrowithcdr-pdf;memmap2in
malmal-cli.
Fixed
pdf.javascriptandpdf.embedded_filewere reported for name trees that list
nothing,pdf.embedded_filefor the packets of an XFA form and for the rights of a
signature (/EFas an array). 428 of the 9,107 clean files of CIC-Evasive-PDFMal2022
were refused for one of these.sanitizeheld the pixels of an image twice while writing it: a PNG at the limits
took 494 MB and takes 250 MB.- A PDF of thousands of streams that all run to one
endstreamwas read once for each
of them: 8 s for a file of 800 KB. The work is now cut off withlimit.complexity,
and the time limit is checked between streams.
Behaviour to check when upgrading
-
A link to a file on the reader's own machine that is not a program (
file:///C|/…)
is no longerpdf.dangerous_uri; a network share or a program still is. -
limit.complexitycounts objects and the values that references lead to, one for
every 16 bytes of the file and two million at least, and no longer every value in the
file. Documents of many objects that were refused are now analyzed. -
The address-space limit of the sandboxed daemon grows with
--workers,--max-size
and--max-decompressedbeyond their defaults. -
malmalandmalmaldexit with status 3 when a scan runs for six times the scan
timeout. Run the daemon under a supervisor that restarts it, or pass--hard-timeout 0. -
PDFs that came back clean may now be suspicious or malicious for what the deeper
checks...