Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: pg_dump support + schema evolution #837

Merged
merged 22 commits into from Feb 13, 2024
Merged

feat: pg_dump support + schema evolution #837

merged 22 commits into from Feb 13, 2024

Conversation

rebasedming
Copy link
Collaborator

@rebasedming rebasedming commented Feb 12, 2024

Ticket(s) Closed

  • Closes #

What

Implements the tableAM scan methods, which are used by the COPY to STDOUT command (invoked by pg_dump). Also handles ADD/DROP/RENAME column correctly.

How to test:

  1. Inside cargo pgrx run, create a deltalake table.
CREATE TABLE t (a int, b text) USING deltalake;
INSERT INTO t VALUES (1, 'test'), (2, 'another');
  1. Exit the connection and run a pg_dump
pg_dump -U <username> -d pg_analytics -h localhost -p 28816 > sample_db_backup.sql

Make sure the replace username with your username. This will create a dump of your database to sample_db_backup.sql.

  1. Reconnect to the database and drop the table you just created.
DROP TABLE t;
  1. Exit the connection and run the backup script
psql -U<username> -d pg_analytics -h localhost -p 28816 -f sample_db_backup.sql
  1. Reconnect to the database to see the dropped table restored.
SELECT * FROM t;

Why

Enables users to back up deltalake tables with pg_dump.

How

Two changes:

  1. Implemented the table AM scan functions, which are used by pg_dump to read from the table.
  2. Fixed the way we were handling ALTER statements on deltalake tables to allow more statements to pass through the Postgres. Also enabled users to run ADD COLUMN over deltalake tables.

Tests

See new tests.

@rebasedming rebasedming changed the title feat: TableAM scan feat: pg_dump support Feb 12, 2024
@rebasedming rebasedming force-pushed the feat/scan branch 2 times, most recently from 6134917 to b2cb1a1 Compare February 12, 2024 18:54
@rebasedming rebasedming marked this pull request as ready for review February 13, 2024 15:49
@rebasedming rebasedming changed the title feat: pg_dump support feat: pg_dump support + schema evolution Feb 13, 2024
Copy link

codecov bot commented Feb 13, 2024

Codecov Report

Attention: 265 lines in your changes are missing coverage. Please review.

Comparison is base (87e7f25) 30.93% compared to head (73c4336) 0.00%.

Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #837       +/-   ##
==========================================
- Coverage   30.93%   0.00%   -30.94%     
==========================================
  Files          84      45       -39     
  Lines        8060    4500     -3560     
==========================================
- Hits         2493       0     -2493     
+ Misses       5567    4500     -1067     
Files Coverage Δ
pg_analytics/src/hooks/executor.rs 0.00% <0.00%> (ø)
pg_analytics/tests/basic.rs 0.00% <0.00%> (ø)
pg_analytics/src/errors.rs 0.00% <0.00%> (ø)
shared/src/fixtures/db.rs 0.00% <0.00%> (ø)
pg_analytics/src/hooks/process.rs 0.00% <0.00%> (ø)
pg_analytics/src/datafusion/datatype.rs 0.00% <0.00%> (ø)
pg_analytics/src/hooks/rename.rs 0.00% <0.00%> (ø)
pg_analytics/src/hooks/alter.rs 0.00% <0.00%> (ø)
pg_analytics/src/datafusion/schema.rs 0.00% <0.00%> (ø)
pg_analytics/src/tableam/scan.rs 0.00% <0.00%> (ø)

... and 45 files with indirect coverage changes

@rebasedming rebasedming merged commit 22794ad into dev Feb 13, 2024
14 checks passed
@rebasedming rebasedming deleted the feat/scan branch February 13, 2024 20:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants