Skip to content

FDW and data source for querying CSV files in S3 and over HTTP#397

Merged
mildbyte merged 16 commits intomasterfrom
feature/csv-s3-fdw
Mar 2, 2021
Merged

FDW and data source for querying CSV files in S3 and over HTTP#397
mildbyte merged 16 commits intomasterfrom
feature/csv-s3-fdw

Conversation

@mildbyte
Copy link
Copy Markdown
Contributor

@mildbyte mildbyte commented Mar 2, 2021

Usage:

$ sgr mount csv csv_schema -o@- <<EOF
{
    "s3_endpoint": "s3.amazonaws.com",
    "s3_bucket": "csv_files",
    "s3_region": "eu-west-1"
    "s3_secure": false,
    "s3_object_prefix": "some_prefix/",
    "autodetect_header": true,
    "autodetect_dialect": true,
    "s3_access_key": "$AWS_ACCESS_KEY_ID",
    "s3_secret_key": "$AWS_SECRET_ACCESS_KEY"
}
EOF

# Query tables in csv_schema

@mildbyte
[wip] Start implementing FDW for discovering and querying CSV files i…
d124a3a 
…n an S3 bucket
@mildbyte
Do some refactoring, get direct HTTP URL querying working (with a pre…
11c32b5 
…-signed S3 URL)
@mildbyte
Add sgr mount csv to the command line, fix some command line docs and…
90eac92 
… misc tests.
@mildbyte
Fix the HTTP/S3 streams getting closed after the first read because w…
2658aeb 
…e were returning from a function (instead of yielding), exiting the context manager.
@mildbyte
Handle some serializers putting empty strings ("") into NULL intege…
465f403 
…r fields by emitting NULLs for empty strings (like we do in `sgr csv import` before passing it to `COPY FROM STDIN`).
@mildbyte
Add a load wrapper to the DataSource class that handles packaging the…
1198105 
… result into an actual image.
@mildbyte
Add loading capabilities to ForeignDataWrapperDataSource (select fr…
9c287b6 
…om FDW -> import into new image)
@mildbyte
Add ability to override the root cert in CSV FDW when using direct ht…
be6c3f8 
…tps (SSL_CERT_FILE envvar to match Minio)
@mildbyte
Catch CSV errors when loading from a URL instead of parsing error res…
2e5762e 
…ponses as CSVs.
@mildbyte
Get Singer load() working
92cd05c
@mildbyte
Add ability to change the S3 region to the FDW; strip out the object …
120ed26 
…prefix so that the table name is just the suffix (still will cause issues with disambiguating table names that are longer than 63 characters)
@mildbyte
Add ability to override the SSL cert bundle using the standard `SSL_C…
fcdc43b 
…ERT_FILE` (for cloud API functions like search etc)
@mildbyte
Respect s3_object in the CSV FDW when doing introspection and only …
1709c77 
…limit it to that one object
@mildbyte
Sample more rows in CSV FDW when doing introspection (100 -> 1000). T…
dc9e207 
…his can mitigate issues with some misjudged types (e.g. numeric -> integer) but still won't help them completely.
@mildbyte
Version bumps
deb8c6b
@mildbyte
Fix CSV tests (object prefix now stripped in suggested table names)
26c8f92
@mildbyte mildbyte force-pushed the feature/csv-s3-fdw branch from 859faec to 26c8f92 Compare March 2, 2021 10:06
@mildbyte mildbyte merged commit 272bcf4 into master Mar 2, 2021
@mildbyte mildbyte deleted the feature/csv-s3-fdw branch March 2, 2021 10:31
mildbyte added a commit that referenced this pull request Mar 4, 2021
@mildbyte
Bump version: 0.2.5 → 0.2.6
369eb91 
  * Fix querying when there are NULLs in primary keys (#373)
  * Data source and foreign data wrapper for querying CSV files in S3 buckets and HTTP (#397)
  * Ctrl+C can now interrupt long-running PostgreSQL queries and stop sgr (#398)
  * Support for updating miscellaneous repository metadata from the `sgr cloud metadata` CLI (#399)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mildbyte