4.4.0

Release highlights

BucketObject.iter_content becomes a real chunked stream on every provider:
it takes a chunk_size, yields chunks sized by that rather than by the
content, and no longer materialises a whole object in memory anywhere.
Separately, two pathological patterns in AWS listing are removed - an image
search that scanned the region's entire public catalogue, and a paginated
call that used the caller's result limit as its transport page size.

Enhancements

• iter_content and save_content accept a chunk_size. It
  defaults to 1 MiB and is settable globally through the iter_chunk_size
  provider config value or the CB_ITER_CHUNK_SIZE environment variable,
  alongside the existing CB_MULTIPART_* knobs. Previously the read size
  was hardcoded and inconsistent - 4 KiB on AWS, 64 KiB on OpenStack Swift,
  the service's own chunking on Azure - and callers had no way to change it.
  The AWS value dated from the 2017 boto2-to-boto3 migration, where a
  BucketObjIterator shim replaced boto2's Key (which read in 8 KiB
  BufferSize chunks); it was never a tuning decision. Reading an HTTP body
  at 1 MiB measures ~12x cheaper per byte than at 4 KiB, and the curve is flat
  from 1 MiB up, so larger defaults would only cost memory. Note that
  save_content was never affected: it copied via shutil.copyfileobj,
  which reads in 64 KiB blocks regardless.
• New aws_page_size configuration value. How many records to request
  from AWS per call while satisfying a list method, defaulting to 500. It is
  a transport setting, distinct from default_result_limit, which bounds
  how many results the caller receives; the two used to be the same number.
  It is clamped to what the service permits for the call in hand, so a value
  outside those bounds is adjusted rather than rejected. AWS-specific
  because it only means anything where the provider walks pages itself:
  GCP, Azure and OpenStack each return a single page plus a continuation
  token and let the caller drive.

Fixes

• The default network is created with the configured default CIDR.
  BaseNetworkService.get_or_create_default passed a hardcoded
  10.0.0.0/16 instead of BaseNetwork.CB_DEFAULT_IPV4RANGE, so setting
  CB_DEFAULT_IPV4RANGE was silently ignored on Azure and OpenStack, which
  inherit the base implementation. AWS and GCP override it and were already
  correct.
• Azure no longer splits object content on newlines. iter_content
  returned an io.RawIOBase wrapper, and iterating a raw stream calls
  readline() - so chunks broke at b"\n" at whatever sizes the content
  happened to dictate, and a blob with no newline in it was buffered whole
  however large it was. It now yields chunk_size chunks read over a single
  connection.
• GCP no longer loads the entire object into memory. iter_content
  returned io.BytesIO wrapped around a full get_media().execute(), so
  streaming a large object cost its full size in RAM (and, being a
  BytesIO, also iterated by line). Content is now fetched as successive
  ranged reads of chunk_size bytes, keeping memory flat at one chunk.
  download_to_file remains the faster path for downloading to disk, as it
  fetches ranges in parallel.
• save_content no longer requires iter_content to return a
  file-like object. It copied with shutil.copyfileobj, which needs a
  .read() that the interface never promised - only Iterable[bytes]. It
  now writes the iterated chunks directly, so a provider returning a plain
  generator works.
• AWSImageService.find no longer scans every public image to run its
  tag search. find(label=...) issues two describe_images calls, one
  filtered on name and one on tag:Name, and neither was scoped by
  Owners. The tag:Name half can only ever match images in the calling
  account - AMI tags are not visible across accounts, so an image owned by
  anyone else cannot satisfy the filter however it is tagged - so omitting
  Owners never widened what it could find. It only made EC2 evaluate the
  filter against the whole regional catalogue: measured in ap-southeast-1,
  10.0s unscoped against 0.1s scoped, for identical single-image results.
  The name half is unchanged and still searches public images, which is
  what most callers want; an explicit owners argument still overrides
  both.
• Paginated AWS calls no longer use the caller's result limit as the
  transport page size. BotoEC2Service._get_paginated_results set
  PaginationConfig={'MaxItems': limit, 'PageSize': limit}, conflating how
  many results the caller wants with how many the service returns per
  request. Against a scan that matches sparsely that walks the collection in
  tiny increments: the same filtered describe_images took 977.6s at a
  page size of 5 and 10.0s at 1000, for one result either way. MaxItems
  still bounds what the caller receives; PageSize is now a full page,
  clamped to whatever bounds the service model declares for the operation
  (DescribeRouteTables permits 100 where most permit more, several require
  at least 5, and falling outside them is a hard InvalidParameterValue).
  Since DEFAULT_RESULT_LIMIT is 50, every paginated AWS call was affected,
  not just filtered searches.

Backward compatibility

chunk_size is optional everywhere, so existing calls keep working. The
AWS return value still exposes read/close as before. The Azure and GCP
return values are now plain generators: code that called .read() on them
must iterate instead, or use save_content/download_to_file.

Pull Requests

• Make iter_content a chunked stream with a configurable chunk size by @nuwang in #343
• Create the default network with the configured default CIDR by @nuwang in #344
• Instrument the cloud suites to find where the AWS time goes by @nuwang in #345
• Fix the two effects behind the AWS suite's runtime: unscoped tag search and transport page size by @nuwang in #346

Full Changelog: v4.3.1...v4.4.0