ARROW-7377: [C++][Dataset] Add ScanOptions::MaterializedFields#6018
ARROW-7377: [C++][Dataset] Add ScanOptions::MaterializedFields#6018fsaintjacques wants to merge 1 commit intoapache:masterfrom
Conversation
fsaintjacques
force-pushed
the
ARROW-7377-simplify-parquet-projection
branch
2 times, most recently
from
7c56e03 to
712a1f4
Compare
bkietz
left a comment
bkietz
left a comment
There was a problem hiding this comment.
LGTM.
As a matter of taste I'd prefer that a helper like this was a non member function (in scanner_internal.h, probably).
cpp/src/arrow/dataset/scanner.h
Outdated
There was a problem hiding this comment.
Non blocking: Maybe eventually this should be an accessor for a (maybe lazily initialized) shared_ptr<unordered_set<string>>
There was a problem hiding this comment.
Hmm, that's not compatible with being a non-member though
There was a problem hiding this comment.
I don't understand what this method does. Can you add a docstring?
There was a problem hiding this comment.
Also, what's the point of having this return an unordered_set? Do you plan to implement unions or intersections of those containers?
There was a problem hiding this comment.
Because it's used as a lookup structure. The ordering and the number of repeats is not important (and should not be exposed). I can still convert it to std::vector.
fsaintjacques
force-pushed
the
ARROW-7377-simplify-parquet-projection
branch
from
712a1f4 to
ee979d1
Compare
|
@fsaintjacques needs a rebase. Appveyor failure is a flight fluke: https://ci.appveyor.com/project/ApacheSoftwareFoundation/arrow/builds/29496245/job/wmfio15805fdjek9#L1526 |
fsaintjacques
force-pushed
the
ARROW-7377-simplify-parquet-projection
branch
from
ee979d1 to
5424b46
Compare
cpp/src/arrow/dataset/scanner.h
Outdated
There was a problem hiding this comment.
I don't understand what this method does. Can you add a docstring?
cpp/src/arrow/dataset/scanner.h
Outdated
There was a problem hiding this comment.
Also, what's the point of having this return an unordered_set? Do you plan to implement unions or intersections of those containers?
A small utility used to simplify parquet's implementation. Will be useful for other file formats.
fsaintjacques
force-pushed
the
ARROW-7377-simplify-parquet-projection
branch
from
5424b46 to
ea64d81
Compare
3 participants
A small utility used to simplify parquet's implementation. Will be useful for other file formats.