Skip to content

v0.9.17

Choose a tag to compare

View all tags
@dforsber dforsber released this 18 Jan 11:47
· 23 commits to main since this release
v0.9.17
2598eb1

🚀 New optional proxyAddress parameter for enabling local S3 Object caching

Add new optional parameter proxyAddress (e.g. http://locahost:3203/). When present the S3 URLs will be replaced with URLs pointing to the proxy address.

With proxyAddress: "http://localhost:3203" set, DuckDB fetches the S3 Objects from e.g. http://localhost:3203/BUCKET/PREFIX address instead of s3://BUCKET/PREFIX.

Discussion about proxy Implementations

The proxy should be transparent proxy towards S3.

An example implementation could for example download S3 Objects to local /tmp folder whenever requested. This reduces S3 API calls, but more importantly makes the DuckDB's Glue Table access like reading local files instead of running lots of round trip requests, which increases performance tenfolds.

Assets 2
Loading