-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Create regular public database snapshots #3
Comments
Will be great to share the results here of how long will take to build the snapshot along with how big the files are. just to left it here, my current |
danksu did this: |
Has Dewi considered using the compressed output format of pg_dump for the ETL dumps? That would avoid the extra step to zip the dump (on Dewi's side) and to unzip (on the restore side). Would save time and disk space when restoring. |
could you provide a dump/restore command to test? |
Sorry, I can't. I've spent two weeks trying to follow various guides and advice on Discord but my Etl instance is still far from functional. The pg_dump man page mentions the -Z flag for compression though, so adding compression should be as easy as adding the Z flag. |
The site is giving a 500 Internal Server Error. :( |
Found this via need to do davetapley/helium-tax#83 ahead of new tax year in USA. Just an idea: I've been using https://duckdb.org/ and its ability to use parquet files for another project (migrating from Postgres for OLAP work) and the file sizes are staggeringly smaller. It gets even more mind-bending when you discover duckdb can reference non-local parquet files (e.g. on GCS) and due to the format can run queries against it without pulling the whole file 🤯 I see the ETL page @dansku built uses https://storage.googleapis.com (assuming that's GCS?), so it might not be too difficult to generate parquet files there instead. Happy to do some work on it, if someone can point me to where http://etl-snapshots.dewi.org/ comes from? |
spillere would know more but I believe they are just Postgres dumps from
the DeWi ETL server, which is private.
If you were able to provide a script or similar to dump to parquet we could
probably run on your behalf, but we can’t provide direct access to the
machine afaik
On Sat, Dec 24, 2022 at 9:34 AM Dave Tapley ***@***.***> wrote:
Found this via need to do davetapley/helium-tax#83
<davetapley/helium-tax#83> ahead of new tax
year in USA.
I see https://etl-snapshots.dewi.org/ is still going, but with 1TB file 😱
Just an idea: I've been using https://duckdb.org/ and its ability to use
parquet files <https://duckdb.org/docs/data/parquet.html> for another
project (migrating from Postgres for OLAP work) and the file sizes are
*staggeringly* smaller.
It gets even more mind-bending when you discover duckdb can reference
non-local parquet files (e.g. on GCS)
<https://duckdb.org/docs/guides/import/s3_import.html> and due to the
format can run queries against it without pulling the whole file
<https://duckdb.org/2021/06/25/querying-parquet.html> 🤯
I see the ETL page @dansku <https://github.com/dansku> built uses
https://storage.googleapis.com (assuming that's GCS?), so it might not be
*too* difficult to generate parquet files there instead.
Happy to do some work on it, if someone can point me to where
http://etl-snapshots.dewi.org/ comes from?
—
Reply to this email directly, view it on GitHub
<#3 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAAAO35IHHKHZ4HNVNHJ5M3WO4CWFANCNFSM45HV2NIA>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
|
Indeed we can do that @davetapley |
Awesome! It's a bit hard to test without access to DeWi ETL server, I just need oracle price history for purposes of davetapley/helium-tax#83, Script:
INSTALL postgres;
LOAD postgres;
-- DeWi ETL server creds here:
CALL postgres_attach('host=HOST dbname=DBNAME user=USER password=PASS');
CREATE TABLE prices (block BIGINT, price BIGINT);
INSERT INTO prices (block, price) SELECT block, price FROM oracle_prices;
EXPORT DATABASE 'parquet_dump' (FORMAT PARQUET); There's probably a way to go straight from Postgres to Parquet, but I figure it'd require a Postgres extension, and I know this approach works and only requires |
this data is quite small |
@dansku nice. Is there anyway we can get that on http://etl-snapshots.dewi.org/ ? Also: Want to try Parquet dump (as shown above) for |
@dansku could you dump that quite a lot of people asking for it over on: which is blocked on: |
hey @davetapley adding the oracle price dump again you can also do tax reporting with hotspotty.net |
Discussed with spillere --
For fast ETL spinup might actual full badgerdb too for ETL rather than simple ledger snapshot. I forget
The text was updated successfully, but these errors were encountered: