[SS-44] Add COPY FROM s3 Docs by patrickwwbutler · Pull Request #35301 · MaterializeInc/materialize

patrickwwbutler · 2026-03-02T15:27:19Z

Updates the documentation for COPY FROM mzsql command to include information and syntax on the new COPY FROM s3 feature.

Motivation

https://linear.app/materializeinc/issue/SS-44/write-user-facing-docs-for-copy-from-s3-statement-csv

Description

Adds a new syntax file for copy from s3/url, adds tab to SQL command reference page, and information on how to use it.

github-actions · 2026-03-02T15:27:29Z

Thanks for opening this PR! Here are a few tips to help make the review process smooth for everyone.

PR title guidelines

Use imperative mood: "Fix X" not "Fixed X" or "Fixes X"
Be specific: "Fix panic in catalog sync when controller restarts" not "Fix bug" or "Update catalog code"
Prefix with area if helpful: compute: , storage: , adapter: , sql:

Pre-merge checklist

The PR title is descriptive and will make sense in the git log.
This PR has adequate test coverage / QA involvement has been duly considered. (trigger-ci for additional test/nightly runs)
If this PR includes major user-facing behavior changes, I have pinged the relevant PM to schedule a changelog post.
This PR has an associated up-to-date design doc, is a design doc (template), or is sufficiently small to not require a design.
If this PR evolves an existing $T ⇔ Proto$T mapping (possibly in a backwards-incompatible way), then it is tagged with a T-proto label.
If this PR will require changes to cloud orchestration or tests, there is a companion cloud PR to account for those changes that is tagged with the release-blocker label (example).

martykulma

Nice! The doc should also enumerate the S3 bucket/object ACLs needed for MZ to read the data.

doc/user/content/sql/copy-from.md

doc/user/data/examples/copy_from_s3.yml

doc/user/content/sql/copy-from.md

maheshwarip · 2026-03-05T15:34:51Z

@kay-kim would you be able to review these today? aiming to launch to customers tomorrow. if you're stretched thin, let me know and I can review

kay-kim · 2026-03-05T16:02:25Z

I'll take a look when I get into the office. Wanted to print out the draft freshness guide, so needed home printer.

kay-kim · 2026-03-05T17:42:32Z

doc/user/content/serve-results/sink/s3.md

+Read*         | [`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)      | Grants permission to retrieve an object from a bucket.
+
+{{< note >}}
+*Read - The `s3:GetObject` Read action is only required if you wish to perform bulk imports into Materialize using [`COPY FROM s3`](/sql/copy-from/).


Q: This sink to s3 using COPY TO s3 page ... Why are we including COPY FROM s3 here?

I figured it wouldn't hurt to add it as this is the main source of info on how to set up AWS connections, and then just added the asterisk to note that it's only required if you want to use COPY FROM. I figured it couldn't hurt?

Heh ... it'll be confusing for users who are sinking to S3 ... because it'll make people go "is this me? Will this tutorial make me do a COPY FROM step later?

It might be that we instead need an ingest data > Bulk copy page. Will let @maheshwarip decide (doesn't actually need to be a blocker but something on the backlog). Yeah ... our connections page is shenanigans.

In discussing, we can do the tutorial when we do parquet and we'll go with just the reference page. As such, I would just revert any changes to this tutorial.

doc/user/content/sql/copy-from.md

doc/user/data/examples/copy_from_s3.yml

doc/user/content/sql/copy-from.md

kay-kim · 2026-03-05T19:58:57Z

doc/user/content/serve-results/sink/s3.md

+Read*         | [`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)      | Grants permission to retrieve an object from a bucket.
+
+{{< note >}}
+*Read - The `s3:GetObject` Read action is only required if you wish to perform bulk imports into Materialize using [`COPY FROM s3`](/sql/copy-from/).


Heh ... it'll be confusing for users who are sinking to S3 ... because it'll make people go "is this me? Will this tutorial make me do a COPY FROM step later?

It might be that we instead need an ingest data > Bulk copy page. Will let @maheshwarip decide (doesn't actually need to be a blocker but something on the backlog). Yeah ... our connections page is shenanigans.

kay-kim · 2026-03-05T19:59:16Z

doc/user/content/sql/copy-from.md

+| Read        | [`s3:GetObject`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html)      | Grants permission to retrieve an object from a bucket.            |
+| List        | [`s3:ListBucket`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html) | Grants permission to list some or all of the objects in a bucket. |
+
+As we are not writing to the bucket, we do not need any write permissions, only read and list.


We don't need this sentence.

kay-kim · 2026-03-05T20:01:18Z

doc/user/content/sql/copy-from.md

+### S3 Bucket IAM Policies
+
+To prepare your S3 bucket for bulk import, follow the instructions in the [Amazon S3 Sink guide](/serve-results/sink/s3),
+but, in your IAM policy, instead allow these actions:


Yeah ... I would just specify that you need to allow these in your IAM policy. Depending on product's answer w.r.t. a tutorial, we can point people to that whenever that's done.

Since no tutorial for now, just say you need to allow the following in your IAM policy.

maheshwarip · 2026-03-05T20:55:28Z

Heh ... it'll be confusing for users who are sinking to S3 ... because it'll make people go "is this me? Will this tutorial make me do a COPY FROM step later? It might be that we instead need an ingest data > Bulk copy page. Will let @maheshwarip decide (doesn't actually need to be a blocker but something on the backlog). Yeah ... our connections page is shenanigans.

I can't reply to the parent comment for some reason. It is odd, but let's not block this PR. we can add the bulk copy page later

maheshwarip · 2026-03-06T14:48:12Z

@kay-kim ready for review!

doc/user/content/sql/copy-from.md

Updates the documentation for `COPY FROM` mzsql command to include information and syntax on the new `COPY FROM s3` feature. ### Motivation https://linear.app/materializeinc/issue/SS-44/write-user-facing-docs-for-copy-from-s3-statement-csv ### Description Adds a new syntax file for copy from s3/url, adds tab to SQL command reference page, and information on how to use it. --------- Co-authored-by: Pranshu Maheshwari <pranshu.maheshwari@materialize.com>

patrickwwbutler added 2 commits March 2, 2026 09:06

begin update copy from docs

4389d8a

update docs, remove railroad and grammer files

aec88c8

patrickwwbutler requested a review from a team March 2, 2026 15:27

patrickwwbutler requested a review from a team as a code owner March 2, 2026 15:27

patrickwwbutler requested a review from kay-kim March 2, 2026 15:27

patrickwwbutler changed the title ~~[SS-44] COPY FROM s3 Docs~~ [SS-44] Add COPY FROM s3 Docs Mar 2, 2026

fix trailing whitespace

a48c804

martykulma reviewed Mar 3, 2026

View reviewed changes

address comments, give instructions on IAM policies

97719ac

patrickwwbutler requested review from DAlperin, martykulma and ublubu March 4, 2026 14:39

comment out parquet info until parquet release

7d9dd37

kay-kim requested changes Mar 5, 2026

View reviewed changes

address comments

97dd795

patrickwwbutler requested a review from kay-kim March 5, 2026 19:56

kay-kim reviewed Mar 5, 2026

View reviewed changes

Responding to PR comments

aa8a76f

kay-kim approved these changes Mar 6, 2026

View reviewed changes

doc/user/content/sql/copy-from.md Show resolved Hide resolved

maheshwarip merged commit d8e89b3 into MaterializeInc:main Mar 6, 2026
10 checks passed

Conversation

patrickwwbutler commented Mar 2, 2026

Motivation

Description

Uh oh!

github-actions bot commented Mar 2, 2026

PR title guidelines

Pre-merge checklist

Uh oh!

martykulma left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

maheshwarip commented Mar 5, 2026

Uh oh!

kay-kim commented Mar 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

maheshwarip commented Mar 5, 2026

Uh oh!

maheshwarip commented Mar 6, 2026

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants