Object storage writes for the export functions, which is not the read path reversed

[ChronicallyJD]

The export functions write to a local path only. Since Parquet consumers generally read
from object storage, an export usually has to be copied somewhere else by hand before
anything can use it.

This is the write half of #393, and it is filed separately because it is a different
problem rather than the same one in reverse.

What exists today

All three take a plain path, and it is a local path:

pgcolumnar.export_arrow(rel regclass, path text)
pgcolumnar.export_parquet(rel regclass, path text)
pgcolumnar.parallel_export_parquet(target regclass, path text, workers int DEFAULT NULL)

The argument is already the right shape. If #393 introduces a URL-dispatched byte source,
these take a byte sink at the same seam, and no caller has to change.

Why it is not just #393 with the arrows reversed

Reads are ranged GETs of an object that already exists and does not change. Writes are
not symmetric with that.

• Multipart upload. Large objects go up in parts, each acknowledged, then completed.
  That is a protocol with state, not a single request.
• Failure leaves debris. An abandoned multipart upload can linger and bill until it
  is aborted, so the failure path has to clean up rather than just report.
• No partial visibility. An object appears whole or not at all, which is a difference
  in what "the file so far" means to a concurrent reader.
• parallel_export_parquet writes a directory of files from several workers. With
  N workers uploading concurrently, "the export succeeded" needs defining when some parts
  completed and others did not. Locally a half-written directory is at least inspectable.
  In a bucket it is a set of objects with no marker saying which run produced them.

That last one is the interesting design question and it does not exist on the read side.

Why it matters beyond convenience

Export is the interoperability story: it is how a columnar table becomes something Spark,
Trino, DuckDB or a warehouse can consume. Those consumers read from buckets. Making the
user stage a local file and copy it up is the step that makes the feature look like a
demo rather than a pipeline.

Suggested shape

Sequenced behind #393, because the credential handling, the retry policy and the URL
dispatch should be built once and shared.

1. Single-object upload for export_parquet and export_arrow, multipart above a size
   threshold, with abort-on-failure.
2. Decide and document what parallel_export_parquet guarantees when some workers fail.
3. Only then consider whether a completion marker or manifest is wanted, which is also
   the point where this starts to touch Review what it would take to support Apache Iceberg #388.

Note

This is not currently recorded anywhere. design/PHASE_G_EXTERNAL_PARQUET_PLAN.md covers
object storage for reads and says nothing about writes, and design/ROADMAP.md does not
mention object storage in either direction. See #395 for that gap.

[jdatcmd]

Plan

Agreed that this is not #393 with the arrows reversed, and the multipart point is the
reason. A ranged GET is stateless and idempotent; a multipart upload is a protocol with
state that can leak billable debris on failure. Those are different problems.

What makes writes harder here specifically

Beyond multipart, three things that are ours rather than S3's:

• We already have a parallel writer. parallel_export_parquet runs N workers. On a
  local filesystem they write parts and the coordinator finishes. On an object store the
  natural mapping is one multipart upload with each worker uploading parts, which means
  the upload id is shared state across backends, and a worker failing after
  uploading parts leaves them owned by an upload only the coordinator can abort.
• No atomic rename. The local path can write to a temp file and rename, so a reader
  never sees a partial object. On S3 the equivalent is completing the multipart upload
  last, which is atomic for that object and is why the write must not be a stream of
  appends to the final key.
• Transaction semantics. An export inside a transaction that later rolls back has
  already put bytes in a bucket. Locally the same is true, so this is not new, but it is
  much more visible when the artifact is remote and shared. Whatever we do, the
  documentation has to say plainly that export is not transactional.

Scope I would propose

Do not do this before #393 lands, and do not do it at the same time. #393 establishes
the dependency decision, the credential model and the privilege gate. If writes are
designed in parallel, those get settled twice and probably differently.

When it does start:

1. Sink seam mirroring Object storage reads for external Parquet, which #388 depends on #393's source seam, chosen by URL, behind the existing path
   argument. Signature unchanged for all three functions, which is the property you
   already identified.
2. Single-object multipart first, export_parquet and export_arrow only. One
   backend, one upload, complete-or-abort with the abort in a PG_FINALLY so a FATAL
   is the only path that can orphan an upload.
3. parallel_export_parquet last, and as a separate decision. It may be right to
   have each worker write its own object rather than share one upload, since Parquet
   consumers read multi-file datasets happily and it removes the shared-state problem
   entirely. That is a design question worth answering before implementing, and it might
   make the parallel case easier than the serial one.

Testable without a network, again

Same requirement as #393: MinIO or equivalent, and specifically the failure paths.
Abort-on-error must be tested by injecting an error mid-upload, because that is the
path that leaks money and it is the one that never gets exercised by accident.

The line I would put in the docs on day one

Export to object storage is not transactional and does not roll back. If that is not
acceptable for a use case, export locally and copy. Saying it up front is cheaper than
a support conversation later.

Yours, sequenced behind #393.

[ChronicallyJD]

Sequenced behind #393, not being worked

Recording so it does not look unowned. The credentials and toolchain decisions on #393
govern this too, and multipart upload should not be designed before the read path picks a
shape.

No work from me until #393 has a direction.