/cockroach

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

importccl: EXPORT CSV #25075

Merged
merged 4 commits into from Apr 26, 2018

Conversation

Reviewers

@mjibson mjibson

@cockroachdb/core-prs core-prs

@cockroachdb/sql-execution-prs sql-execution-prs

@cockroachdb/sql-planning-prs sql-planning-prs

@cockroachdb/distsql-prs distsql-prs

@andreimatei andreimatei

@arjunravinarayan arjunravinarayan

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@dt @cockroach-teamcity @andreimatei @mjibson
@dt
Member

dt commented Apr 25, 2018

This is a re-opening of #24755 after it was closed by a mistaken branch deletion, which unfortunately also prevents re-opening in place.

This adds a DistSQL processor that writes whatever it is given as CSV files to a specified storage location.

This processor can be appended as a sink for arbitrary SELECT queries.

The primary intended use-case is dumping tables as CSV for bulk-export, but it can also be used on any SELECT query, so it could also be used in report generation or other workflows.

Release note (enterprise change): prototype support for EXPORT CSV.

@dt dt requested review from mjibson, andreimatei and arjunravinarayan Apr 25, 2018

@dt dt requested review from cockroachdb/distsql-prs as code owners Apr 25, 2018

@cockroach-teamcity

This comment has been minimized.

Show comment
Hide comment
@cockroach-teamcity

cockroach-teamcity Apr 25, 2018

Member

This change is Reviewable
Member

cockroach-teamcity commented Apr 25, 2018

This change is Reviewable

@dt dt referenced this pull request Apr 25, 2018

Closed

importccl: implement EXPORT csv #24755

@andreimatei

This comment has been minimized.

Show comment
Hide comment
@andreimatei

andreimatei Apr 25, 2018

Member
Member

andreimatei commented Apr 25, 2018

Show outdated Hide outdated pkg/ccl/importccl/exportcsv.go Outdated
@mjibson

This comment has been minimized.

Show comment
Hide comment
@mjibson

mjibson Apr 25, 2018

Member

Oh I just remembered about NULLs. We better add a tests for those.
Member

mjibson commented Apr 25, 2018

Oh I just remembered about NULLs. We better add a tests for those.

dt added some commits Apr 26, 2018

@dt
roachpb: rename NullIf to NullEncoding 
Release note: none.
Verified
This commit was signed with a verified signature.
@dt dt David Taylor
GPG key ID: 01D8438FDFAE4E9A Learn about signing commits
8789715
@dt
sqlrunner: pretty-print both tables on mismatch 
Release note: none.
Verified
This commit was signed with a verified signature.
@dt dt David Taylor
GPG key ID: 01D8438FDFAE4E9A Learn about signing commits
9e7d480
@dt
sql: remove DistLoader wrapper 
I concur with Dan’s year-old TODO: DistLoader doesn’t seem like
a useful abstraction and we’re better off just getting an unwrapped
DistSQLPlanner into the caller’s hands so they can use its public API
without fighting though additional indirection.

Indeed, Ideally I’d like to follow this with exporting more of the
DistSQL API, with the goal of eventually moving LoadCSV method to a
function in importccl defined entirely in terms of the public DistSQL
API, though that will obviously require some refactoring and careful
thinking about where to draw maintainable lines between these packages.

Release note: none.
Verified
This commit was signed with a verified signature.
@dt dt David Taylor
GPG key ID: 01D8438FDFAE4E9A Learn about signing commits
65a7e08

@dt dt requested a review from cockroachdb/core-prs as a code owner Apr 26, 2018

@dt

This comment has been minimized.

Show comment
Hide comment
@dt

dt Apr 26, 2018

Member

Done, added tests with both '' and NULL representations for nulls.
Member

dt commented Apr 26, 2018

Done, added tests with both '' and NULL representations for nulls.
@dt
importccl: implement EXPORT CSV 
This adds a DistSQL processor that writes whatever it is given as CSV
files to a specified storage location.

This processor can be appended as a sink for arbitrary SELECT queries.

The primary intended use-case is dumping tables as CSV for bulk-export,
but it can also be used on any SELECT query, so it could also be used in
report generation or other workflows.

Release note (enterprise change): prototype support for EXPORT CSV.
Verified
This commit was signed with a verified signature.
@dt dt David Taylor
GPG key ID: 01D8438FDFAE4E9A Learn about signing commits
Loading status checks…
cc63a5f
@dt

This comment has been minimized.

Show comment
Hide comment
@dt

dt Apr 26, 2018

Member

bors r+
Member

dt commented Apr 26, 2018

bors r+

craig bot pushed a commit that referenced this pull request Apr 26, 2018

craig[bot] @dt @andreimatei
Merge #25075 #25107 
25075: importccl: EXPORT CSV r=dt a=dt

This is a re-opening of #24755 after it was closed by a mistaken branch deletion, which unfortunately also prevents re-opening in place.

This adds a DistSQL processor that writes whatever it is given as CSV files to a specified storage location.

This processor can be appended as a sink for arbitrary SELECT queries.

The primary intended use-case is dumping tables as CSV for bulk-export, but it can also be used on any SELECT query, so it could also be used in report generation or other workflows.

Release note (enterprise change): prototype support for EXPORT CSV.


25107: sql: cleanup reference to ExecCfg r=andreimatei a=andreimatei

A test was using an InternalExecutor to get an ExecutorConfig, but
there's a better way. Also the old way is about to break.

Release note: None

Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Co-authored-by: Andrei Matei <andrei@cockroachlabs.com>
Loading status checks…
5c6299f
@craig

This comment has been minimized.

Show comment
Hide comment
@craig

craig bot Apr 26, 2018

Build failed (retrying...)

craig bot commented Apr 26, 2018

Build failed (retrying...)

craig bot pushed a commit that referenced this pull request Apr 26, 2018

craig[bot] @dt
Merge #25075 
25075: importccl: EXPORT CSV r=dt a=dt

This is a re-opening of #24755 after it was closed by a mistaken branch deletion, which unfortunately also prevents re-opening in place.

This adds a DistSQL processor that writes whatever it is given as CSV files to a specified storage location.

This processor can be appended as a sink for arbitrary SELECT queries.

The primary intended use-case is dumping tables as CSV for bulk-export, but it can also be used on any SELECT query, so it could also be used in report generation or other workflows.

Release note (enterprise change): prototype support for EXPORT CSV.


Co-authored-by: David Taylor <tinystatemachine@gmail.com>
Loading status checks…
2b7b164
@craig

This comment has been minimized.

Show comment
Hide comment
@craig

craig bot Apr 26, 2018

Build succeeded

craig bot commented Apr 26, 2018

Build succeeded

@craig craig bot merged commit cc63a5f into cockroachdb:master Apr 26, 2018
4 checks passed

4 checks passed

GitHub CI (Cockroach) TeamCity build finished
Details
bors Build succeeded
Details
code-review/reviewable Review complete: 0 of 0 LGTMs obtained
Details
license/cla Contributor License Agreement is signed.
Details

@mjibson mjibson referenced this pull request Apr 27, 2018

Closed

bulkio: bulk export table (or any query) as CSV #24641

@dt dt deleted the dt:export branch Apr 27, 2018

@vivekmenezes vivekmenezes referenced this pull request May 1, 2018

Open

qa: EXPORT CSV #25223

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment