Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add pg_stat_progress_copy view with COPY progress report. #6

Closed
wants to merge 2 commits into from

Conversation

simi
Copy link
Owner

@simi simi commented Jan 1, 2021

example output

first console:
yr=# COPY test FROM '/home/retro/test.csv' (FORMAT CSV);

second console:
yr=# SELECT * FROM pg_stat_progress_copy;
  pid   | datid | datname | relid | bytes_processed | bytes_total |
lines_processed
--------+-------+---------+-------+-----------------+-------------+-----------------
 803148 | 16384 | yr      | 16394 |       998965248 |  1777777796 |
    56730126
(1 row)

benchmark

result

           test             with progress   master (32d6287)   difference
 ------------------------- --------------- ------------------ ------------
  COPY table TO                    46.102             47.499       -1.397
  COPY query TO                    52.168             49.822        2.346
  COPY table TO PROGRAM            52.345             51.882        0.463
  COPY query TO PROGRAM            54.141             52.763        1.378
  COPY table FROM                  88.970             85.161        3.809
  COPY table FROM PROGRAM          94.393             90.346        4.047

setup

DROP TABLE test;
CREATE TABLE test(id int, text varchar(255));
INSERT INTO test (SELECT i, i FROM generate_series(1, 100 000 000) AS i);
VACUUM FULL test;
ANALYZE test;

execute

time bin/psql yr -c "COPY test TO '/home/retro/test.csv' (FORMAT CSV);" # test COPY table TO
time bin/psql yr -c "COPY (SELECT * FROM test) TO '/home/retro/test.csv' (FORMAT CSV);" # test COPY query TO
time bin/psql yr -c "COPY test TO PROGRAM 'gzip > /home/retro/test.csv.gz' (FORMAT CSV);" # test COPY table TO PROGRAM
time bin/psql yr -c "COPY (SELECT * FROM test) TO PROGRAM 'gzip > /home/retro/test.csv.gz' (FORMAT CSV);" # test COPY query TO PROGRAM
time bin/psql yr -c "COPY test FROM '/home/retro/test.csv' (FORMAT CSV);" # test COPY table FROM
time bin/psql yr -c "COPY test FROM PROGRAM 'cat /home/retro/test.csv' (FORMAT CSV);" # test COPY table FROM PROGRAM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
1 participant