Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[query] Rewrite ExportEntriesByCol to stage locally #8613

Merged
merged 2 commits into from
Apr 26, 2020

Conversation

tpoterba
Copy link
Contributor

No description provided.

Copy link
Collaborator

@konradjk konradjk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Works perfectly!!! My understanding of the Scala is rudimentary, but it all seems sensible, and is passing our real-world test.

new OutputStreamWriter(bcFS.value.create(partFolder + "/" + j.toString + extension), "UTF-8")
val filePaths = Array.tabulate(endIdx - startIdx) { j =>
val finalPath = partFolder + "/" + j.toString + extension
val tempPath = bcFS.value.getTemporaryFile("file:///tmp", suffix=Some(extension))
Copy link
Contributor

@akotlar akotlar Apr 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is going to conflict with cotton's fs change I believe, will need to be extension=

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, now it will be createTmpPath, which lives on ExecuteContext

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will tag this as WIP, will be easier for me to rebase than him.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the heads up

@tpoterba tpoterba added the WIP label Apr 23, 2020
@tpoterba tpoterba removed the WIP label Apr 24, 2020
@danking danking merged commit 75aed78 into hail-is:master Apr 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants