Object storage reads for external Parquet, which #388 depends on

[ChronicallyJD]

Our external Parquet support reads local files only. Parquet is usually not on a local
file system, so the feature currently works where the data mostly is not.

Measured on author/main

$ grep -ril "s3\|libcurl\|http" src/*.c src/*.h
src/columnar_parquet_reader.c

The single match is pg_sub_s32_overflow, so there is no object-store code, no HTTP
client and no credential handling anywhere in the tree. The FDW accepts two options,
path and partition_columns, and path is a local path or a glob over one.

Why this is a prerequisite rather than a nice-to-have

Apache Iceberg exists largely because Hive-style tables behaved badly on S3, and several
of its design choices only make sense there. Files are immutable because object stores do
not support in-place modification. A snapshot lists its files explicitly so a reader never
has to list a directory, which on an object store is slow. A commit is an atomic swap of
the metadata pointer, and because object stores have no atomic rename, the catalog exists
to provide that swap.

So #388 depends on this. An Iceberg reader without object storage can read a table
somebody copied to local disk, and not a table as deployed.

The existing note understates the work

design/PHASE_G_EXTERNAL_PARQUET_PLAN.md records the decision:

File access scope: DECIDED (2026-07-23) -- local filesystem first. Object storage
(S3 and S3-compatible such as MinIO, plus GCS/Azure) is a future todo behind the same
path/URL option: an s3://bucket/key path resolves through an object-store reader
while /path/file.parquet reads the local FS. The scan core is unchanged either way
because both just hand it bytes.

The first half is right and the reader really is agnostic. The last sentence is the part
to argue with. A local read is a microsecond-scale pread. An S3 read is a network round
trip of tens of milliseconds. Treating the two as interchangeable byte sources produces
something that works and is unusably slow.

What actually differs:

• Ranged GETs. Read the footer, then only the column chunks the projection needs.
  Parquet's footer-at-the-end layout suits this, but the reader has to ask for byte
  ranges rather than stream a file.
• Concurrency. Latency per request dominates, so chunk fetches want to be in flight
  together. The local path never needed that.
• Retries and backoff. Transient failure is normal, not exceptional.
• Caching. Refetching a footer per query is wasteful in a way it never was locally.
• Credentials. FDW options land in pg_foreign_server in plain text, readable by
  anyone who can select from it. That is a design question before it is a coding one.

Suggested shape

1. A byte-source abstraction behind the reader with the local path as the first
   implementation, so the seam is proven before any network code exists.
2. An S3 and S3-compatible implementation with ranged GETs, retries and concurrency.
   MinIO makes this testable in CI without a cloud account.
3. Credentials, deliberately, with the plaintext problem addressed rather than inherited.
4. GCS and Azure after the seam has carried one implementation.

Steps 1 and 2 are the ones that decide whether this is viable.

Not in scope here

Writing to object storage is a separate issue, because multipart upload and its failure
semantics are a different problem from ranged reads.

[jdatcmd]

Plan

Agreed on the framing, including that #388 depends on this rather than the other way
round. The Iceberg-exists-because-of-S3 argument is the right one and it is worth
restating as a scoping constraint: an Iceberg reader that cannot read object storage
reads a copy of a table, not a table.

The decision that shapes everything else, and I would settle it first

Do we implement the S3 protocol, or link a client? Everything below depends on it,
and it is the one choice that is expensive to reverse.

Against linking (libcurl + an SDK): pgColumnar has no HTTP client, no TLS client and
no credential handling anywhere in the tree today, which your grep establishes. Adding
one adds a dependency to a build that currently needs only codec libraries, and it adds
it to an extension that runs in the backend. Credential handling in particular is a
security surface we do not have at all right now.

Against implementing it: SigV4, retries, redirects, and TLS are not a weekend, and
getting TLS certificate validation subtly wrong is worse than not shipping.

I would link libcurl and write only the S3 request signing on top, as an optional
build (--with-object-storage or detected like HAVE_LIBLZ4), so a build without it
keeps today's dependency set and today's attack surface. That should be argued before
any code.

The seam

One byte source, chosen by URL scheme, behind the existing path option so no caller
changes:

path = '/local/file.parquet'   -> existing local reader
path = 's3://bucket/key'       -> object-store reader

The Parquet reader already needs ranged reads (footer first, then column chunks),
which maps onto HTTP Range requests cleanly. That is fortunate and is the reason this
is tractable: we are not streaming whole objects, we are doing the same ranged access we
already do against a file descriptor.

The interface should therefore be read(offset, length) plus size(), and nothing
else. If the seam ends up wider than that, something is leaking.

What I would insist on in the plan

• No credentials in table options. Options that end up in pg_class.reloptions are
  readable by anyone who can see the catalog. Environment, instance metadata, or a
  server-level configuration file, in that order.
• The privilege model is already there and must be reused. pg_read_server_files
  gates local reads today (Server-file privilege model has drifted: docs claim superuser, two functions require only a role, and the boundary test covers neither #330). A URL read must gate at least as strictly. An
  extension that lets a non-superuser make arbitrary outbound HTTP requests from the
  backend is an SSRF primitive, and that is a bigger change than "reads Parquet from S3".
• Testable without a network. MinIO in a container, or a local HTTP server serving
  byte ranges. A feature that can only be tested against real S3 will not be tested.
  This should be a stated requirement of the design, not discovered later.
• Failure and latency behaviour: a timeout, a retry policy, and a bounded number of
  requests per scan. A ranged reader that issues one request per chunk over a high
  latency link performs very differently from the local path, and the benchmark page
  should say so rather than implying parity.

Sequencing

1. Settle the dependency decision (above), in the issue, before code.
2. Byte-source seam plus the local implementation refactored onto it, with no
   behaviour change and the existing Parquet suites green. That is a safe, reviewable
   first PR that de-risks the rest.
3. S3 reader behind the optional build, tested against MinIO.
4. Only then Review what it would take to support Apache Iceberg #388.

Step 2 is worth doing on its own merits even if the rest is deferred, because it makes
the local path a strategy rather than an assumption.

I have no strong claim on this one; it is squarely yours. My contribution is the
security framing, which I would not want implicit.

[ChronicallyJD]

Suggesting you take the decision, at least

Happy to do the work, but the first call is not a coding one and it is yours.

Two things block a sensible start:

• Build or adopt. An object-store client is where a Rust or C++ toolchain enters our
  build, and we deliberately hand-wrote a Parquet reader to avoid exactly that. That is a
  provenance and build-system decision.
• Credentials. FDW options land in pg_foreign_server in plain text. Whatever we do
  about that is a design decision before it is an implementation.

Decide those two and I will pick up the byte-source seam and the S3 implementation, which
is the part that is just work. #394 is sequenced behind this regardless.