Skip to content

About

Fail-closed upload inspection for Ruby & Rails: blocks disguised executables, Office macros/OLE/DDE/remote templates, PDF JavaScript & launch actions, and zip bombs. Pure Ruby, sandboxed child process, no AV engine.

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Repository files navigation

upload_inspector

Fail-closed structural inspection of untrusted uploaded files for any Ruby or Rails app. Checks a path, a Rails upload, an Active Storage attachment, any IO, or an S3 object uploaded through a presigned URL.

A file is accepted only when its structure is proven to match a known-safe shape. Anything unknown, unparsable, encrypted, oversized, or erroring is rejected. Pure Ruby (marcel, rubyzip), no AV engine, no native parsers, nothing is rendered or executed.

Not an antivirus. It blocks structural attack vectors: disguised executables, macros, PDF JavaScript, OLE objects, remote templates, DDE, zip bombs, image polyglots, scripted SVG and CSV injection. It does not detect malware signatures or exploits hidden inside otherwise valid content. Pair it with an AV scanner (e.g. Amazon GuardDuty Malware Protection for S3, endpoint AV) when you need that.

What it does and does not defend against: THREAT_MODEL.md. Report vulnerabilities privately: SECURITY.md.

What it checks

Format Checks
All real type by magic bytes (Marcel) must be allowlisted; file extension must agree with content; size limit; file name without bidi overrides (invoice\u202Efdp.exe), control characters or an executable extension before the last one (shell.php.jpg, invoice.exe.pdf)
PDF (and PDF-based .ai) rejects /JavaScript /JS /Launch /EmbeddedFile(s) /EF /FileAttachment /Filespec /Collection /RichMedia /XFA /SubmitForm /ImportData /GoToR /GoToE /Rendition, hex-escaped spellings included, also inside compressed object streams (located by /ObjStm or /First and parsed with a real PDF lexer, so comments, strings and fake headers cannot hide them); rejects risky parsers (/JBIG2Decode, 3D/U3D/PRC, PostScript XObjects), URI links other than http(s)/mailto/relative (file:, UNC, javascript:, indirect values), encrypted PDFs and unsupported object-stream filters
docx / pptx / xlsx every part name must match an allowlist, and no check trusts a part's path (Office finds parts through content types and relationships): every part is inspected as XML or proven not to be XML, every XML part gets every check, and only relationship types known to be passive are accepted. Rejects VBA, embeddings/, activeX/, macro sheets, external data (connections, query tables, external/DDE/OLE links, Power Query) by part name, content type, relationship type and element, macro content types, DTDs, external relationships other than http(s)/mailto hyperlinks, Word DDE/INCLUDE(TEXT/PICTURE)/IMPORT/LINK/QUOTE/DATABASE/MACROBUTTON fields, PowerPoint program/macro actions, Excel DDE formulas and defined names, malformed XML, XML that is not UTF-8/ASCII/Latin-1; checks run on entity-decoded text and CDATA
PNG / JPEG / GIF / WebP container walked block by block (nothing is decoded): every length must stay inside the file, declared dimensions within max_image_pixels, and nothing may follow the end of the image (appended scripts or archives). JPEG files holding several images back to back (HDR gain maps, MPF) are accepted; other trailers (motion-photo video, vendor data) are rejected, re-encode those
HEIC / HEIF / AVIF / MP4 / QuickTime box structure walked (nothing is decoded): every box length must stay inside its parent and the boxes must cover the file exactly (no appended data), ftyp must come first and name a brand of the detected type, images need a meta box and video a moov box, declared image (ispe) and track (tkhd) dimensions within max_image_pixels. Files whose brand Marcel does not know are reported as QuickTime and rejected unless the brand is qt ; data hidden inside a box (e.g. a zip in a trailing free box) is not detected
SVG static only: rejects script elements, on* handlers, javascript:/data: links (also set through animation), XHTML/MathML content, stylesheet processing instructions, DTDs other than the standard W3C DOCTYPE, CSS @import/escapes, and any reference outside the file (href, src, CSS url()) except #fragment, embedded `data:image/png
CSV / TSV / TXT valid UTF-8 (BOM allowed; UTF-16 and legacy code pages rejected), no NUL or control characters other than tab/CR/LF, at most max_inflate_size. No HTML, SVG or server-side tags (<script, <html, <svg, <img, <iframe, <?php, <%, ..., any tag with an on*= handler, and at the start the signatures browsers sniff as HTML such as <p>, <div>, <!--; a<b and <b> mid-text are fine), which a sniffing browser would render as a page; allow_text_markup: true turns this off. CSV/TSV also reject cells a spreadsheet would run as a function call or DDE (=HYPERLINK(...), +cmd|..., @SUM(...)), found after any comma, semicolon or tab and inside quotes; negative numbers, phone numbers, @handles and plain arithmetic are accepted. allow_csv_formulas: true turns this off
zip path traversal, symlinks, encrypted entries, duplicates, entry/total size and compression-ratio limits, nested archives; every member extracted (bounded) and inspected with the same policy
Others rejected unless listed in allow_unverifiable (type + extension check only)

Large PDFs can be rejected by chance

PDF names are searched in the raw bytes, compressed image data included, so random bytes that happen to spell a short blocked name (/JS, /3D, /EF) reject a clean file as :pdf_active_content or :pdf_risky_feature. The estimated chance is about 1.2% per MB of compressed stream data: roughly 1 in 8 for a 10 MB PDF, 1 in 2 for 50 MB, 7 in 10 for 100 MB. Text-only PDFs are unaffected. For image-heavy PDFs (scans, .ai artwork) keep uploads to a few MB, or expect to ask users to re-export. Skipping stream data is not an option without following the cross-reference table the way a reader does, because objects can hide in it.

Images, SVG, media and text are verified when their MIME type is in allowed_types (image/png image/jpeg image/gif image/webp image/svg+xml image/heic image/heif image/avif video/mp4 video/quicktime text/csv text/tab-separated-values text/plain). Listing them in allow_unverifiable instead skips these checks.

Install

gem 'upload_inspector'

Requires Ruby >= 3.2.

Usage

Quick start

ALLOWED_TYPES = %w[application/pdf image/png image/jpeg text/csv].freeze

result = UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)

if result.safe?
  # store the file
else
  render json: { error: result.message }, status: :unprocessable_entity
end

Every call returns a Result and never raises on a hostile file:

Method Value
safe? true when the file is accepted
reason why it was rejected, e.g. :pdf_active_content, :extension_mismatch (nil when safe)
message end-user message for the reason, e.g. "is password-protected" (nil when safe)
detail technical hint for logs, not for end users
detected_type the real MIME type, e.g. "application/pdf"
would_reject in mode: :report, the reason the file would have been rejected (nil otherwise)

Only types in allowed_types are accepted (default: PDF only). See What it checks for the types that get a structural check.

What you can pass in

check takes the upload object itself. Rails/Rack uploads and Active Storage carry the client-supplied name; anything else needs filename: (its extension must match the content).

# Controller upload (ActionDispatch::Http::UploadedFile)
UploadInspector.check(params[:file], allowed_types: ALLOWED_TYPES)

# Active Storage attachment or blob (downloaded to a temp file first)
UploadInspector.check(contract.document, allowed_types: ALLOWED_TYPES)

# Path, File, Tempfile, StringIO or any IO
UploadInspector.check("/tmp/upload.bin", filename: "report.pdf", allowed_types: ALLOWED_TYPES)
UploadInspector.check(io, filename: "report.pdf", allowed_types: ALLOWED_TYPES)

Inspection takes up to timeout (60 s by default): check large files in a background job.

Rails validator and job

Both load on first use; the gem does not depend on Rails.

class Contract < ApplicationRecord
  has_one_attached :document

  # Checks files uploaded with this request, before save. Keep it to small files.
  validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES }
end

The error type is the rejection reason (errors.details shows { error: :pdf_active_content }), the message comes from result.message. The validator covers UploadedFiles and attach(io:, filename:). Blobs already in storage (direct uploads, signed blob ids) are skipped, and form strings are never read as server paths. Check those with the job:

# After save, the file is in storage. Unsafe attachments are purged.
UploadInspector::InspectJob.perform_later(contract, "document", allowed_types: ALLOWED_TYPES)

# React to a rejection: subclass and override on_unsafe (called after the purge).
class InspectUploadJob < UploadInspector::InspectJob
  def on_unsafe(record, name, attachment, result)
    UploadMailer.rejected(record, attachment.filename.to_s, result.message).deliver_later
  end
end

Pass purge: false to keep unsafe files. Options are the same keywords as check.

Presets

UploadInspector::Presets holds frozen allowed_types lists of verified types, to combine freely. The default stays PDF only.

Preset Types
PDF PDF
OFFICE docx, xlsx, pptx
DOCUMENTS PDF + OFFICE
IMAGES PNG, JPEG, GIF, WebP, HEIC, HEIF, AVIF
SVG SVG
MEDIA MP4, QuickTime
TEXT plain text, CSV, TSV
UploadInspector.check(file, allowed_types: UploadInspector::Presets::IMAGES + UploadInspector::Presets::DOCUMENTS)

Shadow mode and instrumentation

To roll out on a live app, measure first: mode: :report runs every check but always accepts, and sets result.would_reject to the reason it would have rejected. The validator never adds an error and the job never purges.

validates :document, upload_inspector: { allowed_types: ALLOWED_TYPES, mode: :report }

When ActiveSupport is loaded, every check / check_s3 publishes check.upload_inspector (the event's duration is the check time):

ActiveSupport::Notifications.subscribe("check.upload_inspector") do |event|
  # event.payload: { mode:, reason:, would_reject:, detected_type:, filename_ext:, size: }
  rejected = event.payload[:reason] || event.payload[:would_reject]
  Rails.logger.info("upload_inspector #{rejected || 'safe'} #{event.payload[:filename_ext]} #{event.duration.round}ms")
end

Serving uploads safely

A checked file can still be dangerous if the app serves it so a browser renders it as a page.

  • Serve user files from a separate domain (e.g. usercontent.example.com or the S3/CDN domain), never from the app's own origin, so a rendered file cannot read the app's cookies.
  • Send Content-Disposition: attachment (download, not render) except for raster images and video, and always X-Content-Type-Options: nosniff. A strict CSP with sandbox blocks scripts if a file is rendered anyway. Keep SVG and PDF as attachments: browsers run SVG script when the SVG is opened directly, and Chrome refuses to show PDFs under a sandbox CSP.
  • Never reuse the client's name as is. UploadInspector.safe_filename(name, detected_type:) drops the path, control and bidi characters, inner dots (x.php.jpg → x_php.jpg) and Windows reserved names, caps it at 255 bytes, and takes the extension from the detected type.

safe_headers builds those headers (inline: true is honoured for raster images and video only):

result = UploadInspector.check(path, filename: name, allowed_types: ALLOWED_TYPES)
response.headers.merge!(UploadInspector.safe_headers(result, filename: name, inline: true))
# Content-Type (application/octet-stream unless safe), Content-Disposition,
# X-Content-Type-Options: nosniff, Content-Security-Policy: default-src 'none'; ...; sandbox

Active Storage serves only content_types_allowed_inline inline and forces content_types_to_serve_as_binary (HTML, SVG, ...) to download; keep SVG and text in the latter. For S3, set the headers on the presigned GET:

s3_object.presigned_url(:get, response_content_disposition: "attachment; filename=\"#{safe_name}\"",
                              response_content_type: result.detected_type)

Error messages

result.message gives an end-user message for the reason ("is password-protected", "does not match its file extension", ...). Translate or override any of them under upload_inspector.errors.<reason>, with upload_inspector.errors.default as the fallback:

vi:
  upload_inspector:
    errors:
      default: "chứa nội dung không được phép"
      too_large: "quá lớn"

S3 (presigned URL uploads)

Files uploaded straight to S3 never pass through the app, and the browser chose their Content-Type and size. check_s3 reads the size with HeadObject (oversized objects are rejected without downloading), downloads the object pinned to that ETag, and inspects the bytes. client is any Aws::S3::Client (aws-sdk-s3 is not a dependency of this gem).

s3 = Aws::S3::Client.new

result = UploadInspector.check_s3(
  client: s3,
  bucket: "uploads",
  key: "incoming/abc/report.pdf",
  filename: "report.pdf",  # optional, defaults to the key basename
  version_id: nil,         # optional, pin a version on versioned buckets
  allowed_types: ALLOWED_TYPES,
)

s3.delete_object(bucket: "uploads", key: "incoming/abc/report.pdf") unless result.safe?

S3 errors (missing key, access denied, network) raise, so a job retries rather than deleting a file over an outage. Upload to a quarantine prefix nothing serves, and move the object to its final key only once it is safe: a presigned URL can overwrite the object after the check until it expires (use copy_object with copy_source_if_match and the checked ETag).

Protecting the server from large files

No input is read into Ruby memory: S3 objects and IO are streamed to a temp file, Rails uploads are already on disk (Rack buffers them), and inspection runs in the memory-limited sandbox. Formats like zip, Office and PDF need random access, so the whole file must be on disk. What is left to bound is disk and time:

  • Size before download. check_s3 rejects from HeadObject, Active Storage from byte_size, IO stops at max_file_size + 1 bytes. Set max_file_size to what you actually accept, not the 500 MB default.

  • Refuse oversized uploads at the source. Rails: cap the request body in the web server (nginx client_max_body_size). S3: hand out a presigned POST with a size range; a presigned PUT cannot enforce one.

    post = Aws::S3::Bucket.new("uploads").presigned_post(
      key: "incoming/#{SecureRandom.uuid}/${filename}",
      content_length_range: 1..(20 * 1024 * 1024),
      expires: Time.now + 300,
    )
  • Download time. timeout covers inspection only, not the S3 download. Bound the download on the client you pass in, e.g. Aws::S3::Client.new(http_open_timeout: 5, http_read_timeout: 30, retry_limit: 2).

  • Disk under load. Each concurrent check can use up to max_file_size of temp disk (plus extracted zip members, up to max_zip_total_size). Run checks in a job queue with capped concurrency and size the temp volume for concurrency × max_file_size.

Policy options

Option Default
allowed_types ['application/pdf']
allow_unverifiable []
max_file_size 500 MB
max_zip_entries / max_zip_entry_size / max_zip_total_size 5 000 / 200 MB / 1 GB
max_compression_ratio 100 (entries ≥ 1 MB)
max_inflate_size 50 MB per PDF object stream / OOXML XML part / SVG or text file
max_image_pixels 100 000 000 (declared width × height)
allow_csv_formulas false
allow_text_markup false
timeout / cpu_limit 60 s / 60 s
memory_limit 1 GB (RLIMIT_DATA, Linux only)

Sandbox

By default (sandbox: true) inspection runs in a child Ruby process with CPU, memory (Linux) and wall-clock limits. A crash, hang or memory blow-up kills only the child and yields a rejection (:timeout, :resource_limit, :inspection_failed), so web/job workers are unaffected. The child's temp files live in a directory the parent removes, also when the child is killed. It is resource isolation, not a security boundary (same OS user).

Development

Tests build hostile-shaped samples in memory (no real malware, no AV test signatures). Run them in a container or CI rather than on a workstation with endpoint AV:

docker run --rm -v "$PWD":/src:ro ruby:3.3 \
  bash -c 'cp -r /src /app && cd /app && rm -rf vendor .bundle && bundle install && bundle exec rake test'

CI runs the suite on Ruby 3.2 / 3.3 / 3.4 against rubyzip 2.3 and 3.x. The Rails helper tests use activemodel / activejob from the Gemfile (development only).

test/sample_virus_malware.rb is the hostile-sample corpus: every attack shape with its expected verdict, plus known gaps (skipped until fixed). bundle exec rake samples[DIR] writes the inert samples to disk (default tmp/samples) to cross-check against real readers or other scanners.

License

MIT

About

Fail-closed upload inspection for Ruby & Rails: blocks disguised executables, Office macros/OLE/DDE/remote templates, PDF JavaScript & launch actions, and zip bombs. Pure Ruby, sandboxed child process, no AV engine.

Resources

Security policy

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages