v0.5.0

image_pipe_url

Changed

• Released with image_pipe 0.5.0.

image_pipe

Added

• An [:encode, :search, :probe, :ssimulacra2, :reference] span times
  how long autoquality waits for its comparison image (see
  probe cost spans). The
  default Logger doesn't log it.

Changed

• A JPEG, WebP, or AVIF output without alpha that no earlier step copied to
  memory, such as a plain re-encode, a crop, or a resize, is encoded in full
  before the response starts, instead of being copied to memory first. The
  first byte arrives no later. On Apple Silicon, an 18.7 megapixel JPEG
  re-encode takes 130 ms instead of 200 ms, with half the memory peak, and
  WebP and AVIF outputs of that size take the same time and use 30-40 MB
  less memory. This needs the libvips operation cache off
  (Vix.Vips.cache_set_max(0)), as image_pipe_server sets it. PNG and
  progressive JPEG outputs still stream. A corrupt original still returns
  415. The
  [:transform, :materialize]
  span no longer runs for these outputs, and
  [:encode] times the whole encode.
• EXIF-rotated images use less memory and are faster to resize or crop. On
  Apple Silicon, a full-size 18.7 megapixel EXIF-rotated JPEG uses 40 MB
  less memory, and resizing or cropping an EXIF-rotated image takes 13-24%
  less time. This needs the libvips operation cache off
  (Vix.Vips.cache_set_max(0)), as image_pipe_server sets it.
• A CMYK image, or another one whose ICC profile is converted to sRGB on
  input, is converted after a crop instead of before it. On Apple Silicon,
  cropping a 2000x2000 region from an 18.7 megapixel CMYK JPEG takes 230 ms
  instead of 430 ms, with a memory peak of 49 MB instead of 153 MB.
  Converting the whole image takes 370 ms instead of 500 ms. This needs the
  libvips operation cache off (Vix.Vips.cache_set_max(0)), as
  image_pipe_server sets it.
• Converting an image with an ICC profile other than sRGB, such as a Display
  P3 phone photo, to sRGB at full size takes up to 8% less time and, for an
  18.7 megapixel image on Apple Silicon, uses about 45 MB less memory. This
  needs the libvips operation cache off (Vix.Vips.cache_set_max(0)), as
  image_pipe_server sets it. With the cache on, a metadata edit copies the
  image to memory, as before.
• An image whose alpha channel is opaque everywhere, as in many PNG and
  WebP files, is encoded without the alpha channel. The pixels don't
  change, but a PNG output is saved as RGB instead of RGBA, or as gray
  instead of gray with alpha. On Apple Silicon, an 18.7 megapixel PNG
  output encodes 9% faster and is 9% smaller, and an AVIF output encodes
  7% faster.
• Resizing an image with transparency takes up to 40% less time. On Apple
  Silicon, resizing a 3368x2872 PNG to 800 pixels wide takes 111 ms instead
  of 188 ms, with half the memory peak. Resizing it to 2000 pixels wide
  takes 261 ms instead of 377 ms, with a peak 25 MB higher.
• PNG output is 15-35% smaller and encodes 7-29% faster. A PNG that isn't
  a palette image is saved at zlib compression 3, with the paeth row
  filter unless it is grayscale at bitdepth 1, 2, or 4, instead of
  compression 6 with no filter. A filter set in
  png-options still wins.
• JPEG output is up to 18% smaller, with the same pixels. The encoder
  optimizes the entropy coding for each image, which adds 1 to 4 ms to an
  encode of 1-2 megapixels.
• WebP and AVIF autoquality on outputs of up to 6 megapixels is about 5-10%
  faster, with the same output: 110-260 ms on a 4-core x86 machine and
  30-50 ms on Apple Silicon. The search builds its comparison image while
  it encodes the first quality, instead of before.
• A FileSystem cache's startup scan is several times faster and uses a
  quarter of the memory. On a 12-core machine, a cache of 1M entries is
  scanned in 3.5 s instead of 19 s, with a memory peak of 0.6 GB instead of
  2.4 GB. The first scan after an upgrade that changes the metadata format
  takes 28 s instead of 320 s.
• The startup scan deletes leftover files
  while it counts the entries, so :counting? in the cache's
  usage stays true until the scan
  finishes.
• A hit on a recently served cache entry checks its body's size without
  opening it, which makes a warm GET about 11% faster and a HEAD about
  8% faster. A GET whose body is evicted after that check sends nothing
  and looks the entry up again, which usually generates the image again.
• The cache keeps recently served entries in memory without their
  debug header details, so each takes about
  1.1 KB instead of 1.8 KB, at most about 55 MB by default instead of
  90 MB. A debug request reads its entry from disk.
• A file source checks its path for symlinks that leave the root when it
  checks or reads the file, not on every request. A 304 for an image four
  directories deep is about 10% faster. A request within a
  fallback lifetime, or
  for an immutable source,
  can still be served from the cache after a symlink in its path is changed
  to point outside the root.
• Checking whether to build a
  request summary no longer
  gets slower as more telemetry handlers are attached. With
  ImagePipe.Telemetry.attach_default_logger/1 and
  ImagePipe.Telemetry.attach_tracer/1, it took 4.6 µs per request and
  now takes 0.13 µs.
• A request computes its cache key and ETag in 7 µs instead of 12 µs.
  The cache key is computed differently, so after the upgrade each
  processed image in the cache misses once and is generated again.
  ETags don't change.

Fixed

• An embedded profile that matches sRGB is used as sRGB even when images
  with other profiles are processed at the same time. Before, with the
  libvips operation cache off, as image_pipe_server sets it, such a
  profile could be judged not to match sRGB. Its images were then converted
  through the profile on every request, up to a node restart, and their
  output could differ by 1 out of 255 in a few channel values.
• A response from a source configured to deny storage gets
  Cache-Control: no-store even when an earlier Plug set Cache-Control
  or Set-Cookie. Before, it kept the earlier Plug's caching headers and
  sent no no-store.
• Identical requests for a processed image share one render while up to
  4,096 other images are processing or waiting for a turn. Before, past 64
  such images, each identical request rendered its own copy: a cold burst
  of 160 thumbnails, each requested 3 times, made 342 renders instead of
  160 and took twice as long.
• A streamed image gives back its processing turn when its encode ends,
  instead of when the client has read the last byte. The part of the
  response the client hasn't read yet waits in memory, up to 4 MB, and the
  rest in a file in the system temporary directory. All requests on a node
  together keep about 256 MB in memory and 2 GB in files this way, so the
  temporary directory needs room for 2 GB (see
  slow clients). Before, a few slow
  clients downloading large images could hold every turn, so other requests
  got 503, and identical requests waited for the slowest client.
  While ImagePipe has room to hold what the client hasn't read (see
  deadlines), processing_timeout
  no longer counts the client's download, so a slow download of a large
  image is no longer cut short at the deadline.
• On a CMYK source without a usable embedded ICC profile, colors from the
  request, such as bg, colorize and duotone, come out as requested.
  Before, pad=5/bg=00ff00 painted (192, 208, 11) instead of
  (0, 255, 0), and a progressive blur or several blur groups could
  crash the BEAM.
• An RGB or grayscale source whose embedded ICC profile is for another color
  space, or is damaged, is treated as having no profile. Colors from the
  request come out as requested, and the output doesn't carry that profile.
  Before, the output carried it, a JPEG with RGB pixels and a grayscale
  profile painted pad=5/bg=00ff00 as (220, 220, 220), and one with a
  damaged profile returned 422 for bg, extend and colorize.
• An RGB or grayscale image whose ICC profile only supports the perceptual
  rendering intent, such as a PQ or HLG JPEG XL, is converted through that
  profile. Before, its colors went out unconverted, and bg and colorize
  returned 422.
• An HTTP source that redirects to a protocol-relative Location on the
  same host, such as //images.example.com/moved.jpg, keeps its
  configured auth, aws_sigv4 and headers on the next request. Before,
  ImagePipe treated it as another origin and dropped them, so the origin
  usually answered 401 or 403.
• A file source and the filesystem cache treat a path under their root
  that follows more than 40 symlinks as unsafe. A file source request for
  such a path fails as a denied path (404). Before, some symlink loops
  inside the root, such as one that passes through a link to ., hung the
  request, or the cache's startup check or first lookup, and grew memory
  without bound.
• With stages: true, the default Logger logs :processing_error on
  [:source, :watermark] and [:transform, :execute], and :error on
  [:transform, :operation], at :warning. Before, it logged them at the
  configured level.
• The tracer ends an image_pipe.http.client span with an error status
  when a step of the HTTP request raises an exception. Before, the span
  stayed open and the trace lacked it.
• An HTTP or S3 source whose response declares a Content-Length over
  max_body_bytes fails as soon as the headers arrive. Before, the body
  kept arriving into memory until ImagePipe read past the limit.
• The filesystem cache deletes an entry's temporary metadata file when
  writing it fails, such as on a full disk. Before, the file stayed until
  the daily sweep.
• The filesystem cache stores an image larger than its
  limit on unfinished writes (256 MiB, or a
  twentieth of max_size_bytes when that is smaller) when no other write is
  in progress. Before, it never stored such an image and logged
  stage_dropped (pending_limit) each time, so small caches regenerated
  large images on every request.
• An AVIF whose major brand is a generic HEIF brand, such as mif1, with
  avif among its compatible brands is detected as AVIF. Before, it was
  detected as HEIF, so skip_processing_formats: [:avif] didn't pass it
  through and info reported heif.
• A src or src64 source that isn't valid UTF-8 fails with 400
invalid source before the source is fetched, as a watermark source
  does.
  Before, it reached the source and usually failed with 404.
• In telemetry, a query part without a value, such as &token in a
  source URL or in a request's query string, is replaced with REDACTED.
  Before, it was kept, so a bare token that an origin authorizes with
  appeared in :source_url, the logged path, and traces.