Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sql: support COPY from stdin binary #19603

Closed
maddyblue opened this issue Oct 27, 2017 · 5 comments
Closed

sql: support COPY from stdin binary #19603

maddyblue opened this issue Oct 27, 2017 · 5 comments
Labels
A-disaster-recovery A-sql-pgwire pgwire protocol issues. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)

Comments

@maddyblue
Copy link
Contributor

maddyblue commented Oct 27, 2017

The binary mode is used by https://github.com/jackc/pgx and would be nice to be able to use as an alternative to lib/pq's funky COPY support that mandates a transaction.

From: https://www.postgresql.org/docs/9.2/sql-copy.html

The binary format option causes all data to be stored/read as binary format rather than as text. It is somewhat faster than the text and CSV formats, but a binary-format file is less portable across machine architectures and PostgreSQL versions. Also, the binary format is very data type specific; for example it will not work to output binary data from a smallint column and read it into an integer column, even though that would work fine in text format.

The binary file format consists of a file header, zero or more tuples containing the row data, and a file trailer. Headers and data are in network byte order.

@dianasaur323 dianasaur323 modified the milestone: 1.2 Oct 29, 2017
@knz knz added A-sql-pgwire pgwire protocol issues. A-disaster-recovery C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) labels Apr 28, 2018
@knz knz moved this from Triage to Feature requests / pie-in-the-skie in (DEPRECATED) SQL Front-end, Lang & Semantics Apr 30, 2018
@jordanlewis jordanlewis moved this from Triage to Lower priority backlog in [DEPRECATED] Old SQLExec board. Don't move stuff here May 7, 2019
@rolandcrosby rolandcrosby moved this from Triage to Backlog: Disowned in Disaster Recovery Backlog Jul 10, 2019
@dt dt removed this from Backlog: Disowned in Disaster Recovery Backlog Jul 16, 2019
@asubiotto asubiotto moved this from Lower priority backlog to [TENT] SQL Features in [DEPRECATED] Old SQLExec board. Don't move stuff here Apr 2, 2020
@rafiss
Copy link
Collaborator

rafiss commented Jul 7, 2020

See also #41608 for additional COPY syntax

@solongordon
Copy link
Contributor

Sending to Bulk I/O backlog since we don't have COPY expertise on SQL Features. Would you be able to evaluate how difficult this implementation is? Per @vy-ton we want to shore up features required by pgx so that we can recommend it to customers.

@dt
Copy link
Member

dt commented Jul 20, 2020

@mjibson which usage were you thinking of? the file upload stuff?

@maddyblue
Copy link
Contributor Author

COPY BINARY is just what pgx hard codes as its COPY implementation. Nothing to do with IMPORT or file uploads.

@dt dt removed this from Triage in Disaster Recovery Backlog Jul 22, 2020
@dt
Copy link
Member

dt commented Jul 22, 2020

Ah, okay, so this is about supporting the binary format modifier to the SQL COPY .. FROM statement, which gets planned as an INSERT? Got it -- I don't think that overlaps with any bulk-io stuff then -- we just use COPY support, but didn't implement or maintain it.

vxio pushed a commit to vxio/cockroach that referenced this issue Aug 11, 2020
The COPY binary format is the same as pgwire's binary format. Add a
small state machine and tuple decoder to handle it.

Fixes cockroachdb#19603

Release note (sql change): support BINARY format for COPY FROM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-disaster-recovery A-sql-pgwire pgwire protocol issues. C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
Projects
No open projects
Development

No branches or pull requests

6 participants