v0.3.0

image_pipe_url

Added

• The kernel resize option (resize: [kernel: :nearest] in the builder)
  sets the filter a group's resize uses: nearest, linear, cubic,
  mitchell, lanczos2, lanczos3 (the default), mks2013, or mks2021.
• The priority request option (priority: :low in the builder) lets a
  request wait for spare capacity in the server's processing pool.

Changed

• Breaking: ImagePipe.URL.config/1 raises ArgumentError when a preset
  or the request defaults in :validate_against set options that can't be
  combined, such as anchor=top/detect=face or q=80/autoquality=50. It also
  raises when they name a watermark that the :watermarks list leaves out, or
  use wm-src64 or wm-enc with request_watermarks: false. A server
  running the matching image_pipe release fails at startup with the same
  presets. Remove one of the conflicting options, or the watermark option,
  from the server's presets and the builder's copy.

Fixed

• ImagePipe.URL.new/1 rejects filename: "unset" and cachebuster: "unset"
  instead of building a URL without them. In a URL, unset clears an option.
• ImagePipe.URL.group/2 rejects the detection class "unset", which URLs
  don't accept.
• A float of 2^53 or more, such as rotate: 1.0e300 or a trim tolerance of
  1.0e308, keeps its exact value in the built URL. The URL used to carry a
  nearby whole number instead, so rotate: 1.0e300 built a URL that rotated
  by 280 degrees rather than 0.

image_pipe

Added

• The kernel URL option sets the filter a group's resize uses: nearest,
  linear, cubic, mitchell, lanczos2, lanczos3 (the default),
  mks2013, or mks2021. With kernel=nearest in the first group, a large
  JPEG or WebP source is decoded at full size, so the request takes longer
  and uses more memory. See kernel.
• dns_cache: false turns off an HTTP source's DNS cache, for a server
  whose DNS resolver already caches and whose TTLs should apply exactly.
• max_blur_work limits blur and progressive-blur: the pixels of the
  image being blurred, times the square of the sigma, can't exceed it. A blur
  over the limit fails with 422. See
  Limiting blurs.
• A [:request, :summary] telemetry event, emitted once per request, with
  what the request did as flat fields: method, path, status, duration, bytes
  sent, the signing key, the output and input cache outcomes, source, the
  original's format and size, the operations, the output format and size,
  the quality a search chose, and the time spent fetching, decoding,
  encoding, and waiting for the processing pool. It also records a result
  scaled down to the limits, ignored options, an original sent unprocessed,
  an original held in memory because it couldn't be written to a temporary
  file, and a client that disconnected. It gathers them from every process
  that served the request. The
  event reference lists them.
• The [:request] span's start carries :method and :path for Plug
  requests. The path is as sent, except that a presigned source URL's
  credentials, such as an X-Amz-Signature value, are replaced with
  REDACTED. The tracer records them on the request span as
  http.request.method, url.path, and url.query.
• A file system cache logs at info when it claims its cache root, naming
  the host. A node that refused the root, because another node used it, logs
  at info when it takes the root over and requests use the cache again.
• ImagePipe logs a warning, once per directory, when the image_pipe
  directory in the system's temporary directory can't be used to stage
  originals, so they are held in memory instead. The [:source, :stage]
  span reports each original held in memory, and why, in :staging and
  :staging_reason, which the request summary also carries.
• An S3 source logs a warning when a fetch of its provider's credentials
  fails, naming the bucket and the provider module, and logs at info when a
  fetch succeeds again.
• A priority=low request waits for
  spare capacity in the processing pool: it starts only when no normal
  request is waiting, and a normal request that finds the queue full takes
  its place. The pool's low_priority_max_concurrency option limits how
  many run at once. It defaults to half of max_concurrency, rounded down,
  and at least 1. A low-priority request waits for the original to finish
  downloading before processing starts. See
  Low-priority requests.
• The 503 for a full processing queue or a queue timeout has a
  Retry-After header, the pool's queue_timeout in whole seconds, rounded
  up.
• The [:processing, :admission] and [:processing, :execute] spans carry
  the request's :priority, and ImagePipe.ProcessingPool.stats/1 also
  returns :low_active and :low_queued. The [:request, :summary] event
  has priority: :low for a low-priority request, and the default Logger
  logs one that a full pool turned away at its base level instead of as an
  error.

Changed

• An instance without cache or input_cache no longer reads and hashes an
  unchanged local file on every request. It keeps the hashes of up to 10,000
  files in memory and checks each file's size, timestamps, inode, and device
  instead. For a 5.5 MB PNG this saves about 3 ms per request.
  For a remote original, a request with If-None-Match gets 304 without
  contacting the origin while the origin's cache lifetime lasts.
  [:cache, :lookup] reports these
  lookups with pool: :memory.
• An HTTP source keeps the addresses a hostname resolves to for 10 seconds,
  so a busy origin is no longer looked up before every fetch. The addresses
  are still checked against :address_policy on every fetch. A resolver
  passed in :address_resolver replaces the cache.
• When none of an HTTP source's addresses accepts a connection or answers in
  time, the source looks the hostname up again and tries once more if the
  answer changed. A resolver passed in :address_resolver can therefore be
  called twice for one fetch.
• AVIF's default quality is 56, down from 63. Files are about 18% smaller.
  The median image scores 78 instead of 82 on
  SSIMULACRA2,
  close to the JPEG default's 77. To keep 63, set
  format_quality: %{avif: 63}.
• A response's cache key and ETag no longer depend on the quality and
  encoder settings of other formats. Changing the AVIF quality or
  avif_options leaves them unchanged for WebP responses and for requests
  with a format, such as format=jpeg, so those stay cached and clients
  keep their copies. A response without format, for a client that accepts
  neither AVIF nor WebP, takes the original's format and still depends on
  every format's settings. After upgrading, cached image responses miss once
  and clients download each image again once.
• Breaking: a blur or progressive-blur over the default
  max_blur_work, 4,000,000,000,000, fails with 422. blur=1000 on a 10-megapixel photo took about 7 seconds
  and now fails. Raise max_blur_work to allow larger blurs.
• blur and sharpen buffer the image in memory before filtering it. They
  recomputed earlier steps, such as an enlarge or a color conversion, many
  times over, so sharpen=10/w=8000/h=8000/enlarge took 14 s. It now takes
  under 1 s. A request with blur or sharpen holds one copy of the image in
  memory at that point.
• Cached output=blurhash, output=lqip-css, and output=info responses
  that include a placeholder miss once after upgrading, since placeholders of
  transparent images changed.
• Output cache write failures are no longer logged directly. Attach the
  default Logger to see them, as image_pipe cache stage and
  image_pipe cache write warnings with the error category.
• [:cache, :write] carries :error on input cache write errors too.
• Breaking: ImagePipe.Telemetry.attach_default_logger/1 logs one line
  per request, from the new [:request, :summary] event, instead of one line
  per stage. A 5xx, or a response that failed partway through sending,
  logs at error, and a 4xx, a result that didn't come out as requested,
  or a failed cache read at warning. Failed and dropped cache writes log
  their own warning lines. Pass stages: true to also log the stage lines
  as before.
• Breaking: An ImagePipe instance starts a processing pool unless its
  configuration names one in processing_pool. It processes as many images
  at once as the VM has online schedulers and queues up to 64 more for up to
  10 seconds. Beyond that, requests get a 503, and ImagePipe.run/4
  returns {:processing, :overloaded} or {:processing, :queue_timeout}. An
  image that takes longer than 30 seconds gets a 503, or
  {:processing, :timeout} from run/4. Set the new pool instance option
  to change the limits, or pool: false to process every request as it
  arrives, as before.
• Breaking: ImagePipe.config/1 raises ArgumentError for a :detector
  module that isn't loaded or doesn't implement
  ImagePipe.Transform.Detector. Before, such a configuration started, and
  every detect or anchor=smart-face request failed with 500. Fix the
  module name, or set detector: nil.
• Breaking: ImagePipe.config/1 raises ArgumentError for a preset or
  request defaults whose own options can't be combined, such as
  anchor=top/detect=face or q=80/autoquality=50. It also raises for a wm
  name missing from :watermarks, and for wm-src64 or wm-enc without
  request_watermarks: true. Requests that used them failed with 400
  unless the URL replaced the conflicting options. Remove one of the
  conflicting options, or the watermark option.
• A preset from a preset lookup whose own options can't be combined answers
  500, like other invalid stored presets, instead of 400, and
  ImagePipe.run/4 returns {:preset, :invalid_definition} for it.
• A request with pad=0 gets the cache key and ETag of the same request
  without pad, and a gradient or progressive-blur that starts or stops
  at -0.0 gets those of one at 0. Cached copies of such requests miss once
  after an upgrade.
• A format-q entry for png applies only when PNG has a palette, like q.
  Before, a URL with one and no palette failed with 400, even when the
  response wasn't PNG, and one in request_defaults or a preset made every
  request fail.
• A watermark on an image with a color profile is about 5 ms faster when the
  output is sRGB, which is the case with the default profile=strip and with
  profile=srgb. Partly transparent watermark pixels over colors outside
  sRGB, such as on a Display P3 photo, blend slightly differently.

Fixed

• autoquality scored images with transparency as if they were shown on
  black, so changes at their edges and in their transparency didn't count,
  and transparent margins raised the score. A logo could get AVIF quality 22
  where it needed 31. It now scores the visible area only, against both a
  dark and a light background, and keeps the lower score. Images with
  transparency can get a higher quality and larger files.
• Detection requests that arrived before the bundled face or object model
  had loaded, for example during startup warmup with
  detector_required: false, each downloaded and loaded the model. They now
  wait for one load, so the 175 MB object model downloads once.
• png-options=bitdepth:16 multiplied 8-bit color by 256 instead of 257, so
  128 became 32768 instead of 32896 and only white reached 65535. Color now
  scales by 257, as gray already did.
• output=info reported dpr from before the scale-down to the result
  limits, so dividing its width by dpr gave less than the CSS width. dpr now includes that scale-down: w=4000/dpr=3 on a
  9,000-pixel-wide image with an 8,192-pixel limit reports width 8192 at
  dpr 2.048, not 2.25.
• Content-aware crops with detect or anchor=smart-face for a small
  output detected on the resized image. w=64/h=64/fit=cover on a
  6000×4000 source gave the detector 96×64 pixels, so small faces and
  objects were missed. Detection now gets at least 640 pixels on the long
  edge, or the whole source when it's smaller, and such requests decode at
  that size.
• An extend-offset that landed exactly on a half pixel, such as
  extend-offset=0,1.25 with dpr=2, rounded away from zero, placing the
  image 1 pixel off. It now rounds ties to even, as crops, watermarks, and
  padding do.
• With a processing pool, a streamed response that finished just after
  processing_timeout, before the pool's timer fired, ended with
  {:session, {:producer_down, :normal}} instead of
  {:processing, :timeout}.
• output=blurhash and output=lqip-css rendered transparency on black, so
  a transparent PNG's placeholder could be dark while the JPEG the same URL
  returns is light. Placeholders now render transparency on white, unless
  bg sets a background.
• An ImagePipe.Plug configured at compile time, such as in a router's
  forward options, could fail to compile with "does not implement
  ImagePipe.Transform.Detector" when its custom detector was compiled in the
  same pass. Configuration now waits for the detector module to compile.
• The [:transform, :detect, :model] span reported a model that is
  unavailable, such as a face model without image_vision, as result: :error, and the default Logger logged it as a model error. It now reports
  result: :unavailable, as [:transform, :detect] does.
• Watermarks, bg, colorize over transparent areas, JPEG output of an
  image with transparency, and resizing or rotating an image with
  transparency could make a color one level darker than the exact result,
  such as 169 for 169.51. These now round to the nearest level.
• format=png with q, from the URL, the request defaults, or a preset,
  failed with 400 when PNG had no palette. A q in request_defaults or a
  preset made every format=png request fail. PNG without a palette now
  ignores q, as negotiated PNG already did.
• On a URL with expires, a response gets Cache-Control: no-store, even
  when an earlier Plug set Cache-Control, if storage is denied by its origin
  or by configuration, or if a crop fell back because content detection
  failed. Before, it got the earlier Plug's Cache-Control capped at the
  expiry, and a CDN could store it.
• An SVG source with fit=stretch and one axis, such as w=480/fit=stretch,
  or with only min-w or min-h, stayed at its intrinsic size instead of
  rendering at the requested size. It now renders at that size, like any
  other first-group resize of an SVG.
• On a URL with expires, a Cache-Control set by an earlier Plug is capped
  at the expiry in more cases:
  • A quoted value, such as max-age="86400", is read as seconds. A
    max-age that isn't a whole number of seconds, such as
    max-age=86400.5, is replaced, and an s-maxage like that is dropped.
    Before, both failed the request with 500.
  • Directive names match in any case, so MAX-AGE=86400 is lowered too.
  • A Cache-Control without max-age, such as public, gets one that ends
    at the expiry.
• A crop or region followed by a resize could give a different size for
  a JPEG source than for the same image as PNG, because JPEG decodes at a
  smaller size first. A narrow crop was off by several pixels, such as
  crop=101,1500/w=50 giving 50×750 instead of 50×743. The size is now the
  same for every source format, in the image and in output=info.
• A server error (5xx), such as a 502 for a failing origin, gets
  Cache-Control: no-store, even when an earlier Plug set Cache-Control.
  Before, it kept the earlier Plug's Cache-Control, and a CDN could serve
  the error for that header's lifetime. A client error (4xx) still keeps
  it.
• output=info with an explicit format whose encoder has a lower size limit
  than the server's result limits, such as format=webp with limits above
  16,383 pixels, shared its cache entry and ETag with other formats. A cache
  could then serve one format's result dimensions for another.
• A request to an HTTP or S3 source that waited past pool_timeout for a
  free connection crashed the request process. Behind Bandit, the client got
  no response and the connection stayed open. Such requests now fail with a
  {:source, :pool_timeout} error, which the Plug answers with 504.
• autoquality together with max-bytes could ship a lower quality than
  max-bytes alone, because the quality search used up the encodes the byte
  budget needed. The byte budget now gets its own encodes.
• A conditional header in an HTTP or S3 source's req_options, such as
  if-none-match, made the request fail when the origin answered a first
  fetch with 304. Sources now drop if-none-match, if-modified-since,
  if-match, if-unmodified-since, and if-range from req_options.
• The detector warmup gave up after three failed attempts, about a third of
  a second, so with detector_required: true detection requests answered
  503 until a restart. It now retries, at most a minute apart, until the
  models load, and logs at info when they do.
• Req's :range option in req_options set a range header even on sources
  that drop range headers, which is every source except one with
  internal_cache: :disabled that isn't immutable. The origin then sent part
  of the image, which was processed as the whole original. These sources now
  drop the :range option too.
• A watermark whose stored copy in the input cache was corrupt made every
  request using it fail with 415 until the copy was evicted. The copy is now
  discarded after the first failure, and the next request fetches the
  watermark again.
• An HTTP source URL whose query has a space or a non-ASCII character, such
  as ?a b, is fetched with those characters percent-encoded. Before, the
  request failed with 502.
• png-options with bitdepth:2 or bitdepth:4 wrote wrong gray levels,
  such as gray 150 coming back as 102 at 4 bits. A bitdepth of 1, 2,
  or 4 on a result with alpha or color failed with 500. Grayscale
  results without alpha now keep their gray levels, and other results
  without a palette encode at 8 bits.
• A detect, anchor=smart, or anchor=smart-face crop whose box covers
  the whole image, such as crop=8000,8000/detect=face on a smaller image,
  ran the detector or saliency anyway. With a failing detector the response
  wasn't stored, or failed with 500 under detector_required: true. Such
  a crop now runs no detection.
• An HTTP source could run past fetch_timeout in two ways. Looking up the
  host name could take as long as the address_resolver allowed. A name
  that resolved to several addresses made the fetch wait for a response at
  each address in turn. Fetches now fail with 504 at fetch_timeout,
  except that a connection attempt can run past it by up to
  connect_timeout.
• A FileSystem cache could hold more than max_size_bytes on disk and not
  bring it back down:
  • A read of an entry the startup scan hadn't counted yet counted it
    without evicting. The cache now evicts back under its cap.
  • A startup scan that crashed skipped evicting down to a lowered cap. It
    now evicts as a completed scan does.
  • A write during the startup scan that was rejected or failed left its
    key's existing entry on disk uncounted. A write that replaced such an
    entry left its old body on disk. Both are now counted or deleted.
  • An entry written again after the startup sweep removed its unreadable
    metadata, as after an upgrade that changes the metadata format, wasn't
    counted.
• A request that reached processing_timeout while its source was still
  downloading failed with 500 or got no response. It now fails with
  {:processing, :timeout}, which the Plug answers with 503.
• A large pixelate block size, such as pixelate=2000 on a 4000×2667
  JPEG, failed with 415. It now works.
• A region whose width or height came to less than half a pixel, such as
  region=0,0,0.3,1, failed with 500. It now keeps one pixel.
• With detector_required: true, an anchor=smart-face crop called the
  detector before its models were ready, so the built-in detector could
  download a model during the request. Until the face model is ready, such
  a crop now uses attention cropping without calling the detector, and the
  response isn't stored while the face model can still become ready.
• ImagePipe.Source.S3.ContainerCredentials accepted an auth_token with a
  line break or NUL, and every credential fetch then failed. Such a token is
  now rejected when the source is configured.
• A region that started inside the last pixel column or row, such as
  region=2002.6,0,10,10 on a 2003-pixel-wide image or
  region=99.99pct,0,10,10, failed with 400 because its origin rounded
  past the edge. It now starts at the last pixel. Only a region that starts
  at or beyond the edge is rejected.
• output=info reported a result size for a URL whose image request fails
  with 422 because it exceeds max_intermediate_pixels, such as
  w=20000/enlarge on a small image. It now fails with 422 too, as
  output=blurhash and output=lqip-css already did.
• A request for another variant of a source that was still downloading, such
  as w=50 while w=100 was fetching the same image, could fail with 504.
  It waited until the first request's processing, which can start before the
  download finishes, was done too, and gave up a second after the source's
  fetch_timeout. It now waits only for the download.
• An SVG source ignored page, so page=1 on an SVG answered 200 with the
  only image, and each page value stored its own copy. A page past 0
  now fails with 422, as it does for any other single-page image.
• A FileSystem cache that restarted while a write was stuck on disk for more
  than 25 seconds, as on a hung network mount, kept counting the bytes the
  response still sent to that write as waiting to be written. Until the next
  restart, the cache dropped more writes and evicted more entries than it
  needed to. Those bytes are no longer counted.
• A FileSystem cache that was full and busy with other writes could turn
  away an original's new expiry time or content identity after checking the
  original with the origin, and keep the outdated ones. Requests then checked
  with the origin, or downloaded the changed original, every time until the
  cache stored the update. The cache now removes the outdated details when it
  turns an update away.