Iceberg Bucket support #720

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

fenos merged 1 commit into master from feat/iceberg-catalog

Jul 11, 2025

Contributor

fenos commented Jul 9, 2025

What kind of change does this PR introduce?

Feature

What is the new behavior?

Implement a multi-tenant iceberg catalog proxy, currently supported S3Tables catalog

Additional context

Add any other context or screenshots.

snyk-io bot commented Jul 9, 2025 •

edited

Loading

⛔ Snyk checks have failed. 11 issues have been found so far.

Icon	Severity	Issues
	Critical	0
	High	11
	Medium	0
	Low	0

⛔ code/snyk check is complete. 11 issues have been found. (View Details)

Up to 10 code/snyk issues appear as inline comments below; view the rest through the details page.

💻 Catch issues earlier using the plugins for VS Code, JetBrains IDEs, Visual Studio, and Eclipse.

snyk-io bot reviewed

View reviewed changes

src/http/routes/iceberg/namespace.ts

    
                      parent: request.query.parent,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 136 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 22 steps

Step 1 - 6

storage/src/http/routes/iceberg/namespace.ts

Line 126 in a232574

id: request.params.prefix,

Step 7 - 11 src/http/routes/iceberg/namespace.ts#L124

Step 12 - 15 src/http/routes/iceberg/namespace.ts#L130

Step 16 - 20 src/http/routes/iceberg/namespace.ts#L129

Step 21 - 22

storage/src/http/routes/iceberg/namespace.ts

Line 136 in a232574

return response.send(result)

src/http/routes/iceberg/namespace.ts

    
                      namespace: request.params.namespace,

                    })

                    return response.status(204).send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 159 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 13 steps

Step 1 - 6

storage/src/http/routes/iceberg/namespace.ts

Line 156 in a232574

namespace: request.params.namespace,

Step 7 - 11 src/http/routes/iceberg/namespace.ts#L155

Step 12 - 13

storage/src/http/routes/iceberg/namespace.ts

Line 159 in a232574

return response.status(204).send(result)

src/http/routes/iceberg/namespace.ts

    
                      namespace: request.params.namespace,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 182 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 13 steps

Step 1 - 6

storage/src/http/routes/iceberg/namespace.ts

Line 179 in a232574

namespace: request.params.namespace,

Step 7 - 11 src/http/routes/iceberg/namespace.ts#L178

Step 12 - 13

storage/src/http/routes/iceberg/namespace.ts

Line 182 in a232574

return response.send(result)

src/http/routes/iceberg/table.ts

    
                      namespace: request.params.namespace,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 317 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 22 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 308 in a232574

id: request.params.prefix,

Step 7 - 11 src/http/routes/iceberg/table.ts#L306

Step 12 - 15 src/http/routes/iceberg/table.ts#L313

Step 16 - 20 src/http/routes/iceberg/table.ts#L311

Step 21 - 22

storage/src/http/routes/iceberg/table.ts

Line 317 in a232574

return response.send(result)

src/http/routes/iceberg/table.ts

    
                      pageToken: request.query.pageToken,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 343 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 22 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 333 in a232574

id: request.params.prefix,

Step 7 - 11 src/http/routes/iceberg/table.ts#L331

Step 12 - 15 src/http/routes/iceberg/table.ts#L337

Step 16 - 20 src/http/routes/iceberg/table.ts#L336

Step 21 - 22

storage/src/http/routes/iceberg/table.ts

Line 343 in a232574

return response.send(result)

src/http/routes/iceberg/table.ts

    
                      table: request.params.table,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 369 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 22 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 360 in a232574

id: request.params.prefix,

Step 7 - 11 src/http/routes/iceberg/table.ts#L358

Step 12 - 15 src/http/routes/iceberg/table.ts#L364

Step 16 - 20 src/http/routes/iceberg/table.ts#L363

Step 21 - 22

storage/src/http/routes/iceberg/table.ts

Line 369 in a232574

return response.send(result)

src/http/routes/iceberg/table.ts

    
                      table: request.params.table,

                    })

                    return response.status(204).send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 393 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 13 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 389 in a232574

namespace: request.params.namespace,

Step 7 - 11 src/http/routes/iceberg/table.ts#L388

Step 12 - 13

storage/src/http/routes/iceberg/table.ts

Line 393 in a232574

return response.status(204).send(result)

src/http/routes/iceberg/table.ts

    
                        table: request.params.table,

                      })

                      return response.status(204).send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 426 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 13 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 422 in a232574

namespace: request.params.namespace,

Step 7 - 11 src/http/routes/iceberg/table.ts#L421

Step 12 - 13

storage/src/http/routes/iceberg/table.ts

Line 426 in a232574

return response.status(204).send(result)

src/http/routes/iceberg/table.ts

    
                      table: request.params.table,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from an HTTP parameter flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 452 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 13 steps

Step 1 - 6

storage/src/http/routes/iceberg/table.ts

Line 448 in a232574

namespace: request.params.namespace,

Step 7 - 11 src/http/routes/iceberg/table.ts#L446

Step 12 - 13

storage/src/http/routes/iceberg/table.ts

Line 452 in a232574

return response.send(result)

src/http/routes/iceberg/namespace.ts

    
                      warehouse: bucket.id,

                    })

                    return response.send(result)

snyk-io bot Jul 9, 2025

Cross-site Scripting (XSS)

Unsanitized input from the HTTP request body flows into send, where it is used to render an HTML page returned to the user. This may result in a Cross-Site Scripting attack (XSS).

Line 110 | CWE-79 | Priority score 847 | Learn more about this vulnerability

Data flow: 14 steps

Step 1 - 7

storage/src/http/routes/iceberg/namespace.ts

Line 106 in a232574

namespace: [request.body.namespace],

Step 8 - 12 src/http/routes/iceberg/namespace.ts#L105

Step 13 - 14

storage/src/http/routes/iceberg/namespace.ts

Line 110 in a232574

return response.send(result)

coveralls commented Jul 9, 2025 •

edited

Loading

Pull Request Test Coverage Report for Build 16221734173

Details

3012 of 3978 (75.72%) changed or added relevant lines in 64 files are covered.
84 unchanged lines in 10 files lost coverage.
Overall coverage decreased (-0.9%) to 77.093%

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
src/http/plugins/signature-v4.ts	0	1	0.0%
src/http/routes/s3/router.ts	20	21	95.24%
src/http/plugins/tenant-feature.ts	1	3	33.33%
src/http/routes/admin/migrations.ts	0	2	0.0%
src/http/routes/iceberg/catalog.ts	49	51	96.08%
src/internal/database/tenant.ts	38	40	95.0%
src/internal/queue/event.ts	29	31	93.55%
src/storage/events/migrations/run-migrations.ts	1	3	33.33%
src/storage/limits.ts	14	16	87.5%
src/test/utils/storage.ts	40	43	93.02%

Files with Coverage Reduction	New Missed Lines	%
src/http/error-handler.ts	1	59.48%
src/storage/protocols/s3/byte-limit-stream.ts	1	91.3%
src/http/routes/s3/index.ts	2	88.31%
src/http/routes/s3/commands/head-object.ts	3	75.0%
src/http/routes/s3/router.ts	4	94.15%
src/storage/protocols/s3/signature-v4.ts	4	77.71%
src/http/routes/s3/commands/put-object.ts	7	85.43%
src/storage/storage.ts	13	73.72%
src/storage/database/knex.ts	22	83.29%
src/http/plugins/signature-v4.ts	27	44.52%

Totals
Change from base Build 16055934289:	-0.9%
Covered Lines:	20323
Relevant Lines:	26130

💛 - Coveralls

itslenny reviewed

View reviewed changes

migrations/tenant/0038-iceberg-catalog-flag-on-buckets.sql Outdated Show resolved Hide resolved

src/config.ts Show resolved Hide resolved

src/http/routes/admin/tenants.ts Outdated Show resolved Hide resolved

src/http/routes/iceberg/catalog.ts Outdated Show resolved Hide resolved

src/http/routes/s3/commands/head-object.ts Outdated Show resolved Hide resolved

src/http/routes/s3/commands/upload-part.ts Outdated Show resolved Hide resolved

src/storage/database/knex.ts Outdated Show resolved Hide resolved

fenos force-pushed the feat/iceberg-catalog branch from a232574 to 3a9c533 Compare

July 10, 2025 10:08

itslenny approved these changes

View reviewed changes

fenos force-pushed the feat/iceberg-catalog branch from 3a9c533 to e9705ee Compare

July 10, 2025 16:49

fenos commented

View reviewed changes

src/internal/queue/event.ts Outdated Show resolved Hide resolved

fenos force-pushed the feat/iceberg-catalog branch 4 times, most recently from 514e3f3 to e7db982 Compare

July 11, 2025 13:51


          fix: header validation always string (#718)

03c59eb

fenos force-pushed the feat/iceberg-catalog branch from e7db982 to 03c59eb Compare

July 11, 2025 13:52

itslenny reviewed

View reviewed changes

src/storage/protocols/iceberg/catalog/rest-catalog-client.ts

    
                    .then((response) => response.data)

                    .catch((error) => {

                      if (error instanceof AxiosError) {

                        console.error('Error fetching configuration:', error.response?.data)

Contributor

itslenny Jul 11, 2025

Several console.error and one console.log in this file. These should probably use logSchema

fenos merged commit 345aea5 into master

1 of 2 checks passed

fenos deleted the feat/iceberg-catalog branch

July 11, 2025 15:16

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet