Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: Read object store files in parallel #1381

Merged
merged 5 commits into from
Apr 6, 2023
Merged

feature: Read object store files in parallel #1381

merged 5 commits into from
Apr 6, 2023

Conversation

karolisg
Copy link
Contributor

@karolisg karolisg commented Apr 5, 2023

Closes #934

@karolisg karolisg marked this pull request as ready for review April 5, 2023 06:29
@coveralls
Copy link

coveralls commented Apr 5, 2023

Pull Request Test Coverage Report for Build 4626194253

  • 50 of 55 (90.91%) changed or added relevant lines in 3 files are covered.
  • 242 unchanged lines in 6 files lost coverage.
  • Overall coverage increased (+0.2%) to 72.223%

Changes Missing Coverage Covered Lines Changed/Added Lines %
dozer-ingestion/src/errors.rs 0 1 0.0%
dozer-ingestion/src/connectors/object_store/table_reader.rs 46 50 92.0%
Files with Coverage Reduction New Missed Lines %
dozer-ingestion/src/errors.rs 15 7.89%
dozer-ingestion/src/connectors/object_store/table_reader.rs 18 78.77%
dozer-sql/src/pipeline/expression/tests/comparison.rs 23 80.26%
dozer-api/src/generator/protoc/generator/implementation.rs 36 86.67%
dozer-core/src/dag_impl.rs 63 74.8%
dozer-types/src/helper.rs 87 75.7%
Totals Coverage Status
Change from base Build 4614393274: 0.2%
Covered Lines: 33565
Relevant Lines: 46474

💛 - Coveralls

@karolisg karolisg requested a review from chubei April 6, 2023 02:41
@karolisg karolisg merged commit 455e928 into getdozer:main Apr 6, 2023
@karolisg karolisg deleted the feat/paralel-table-reads branch April 6, 2023 07:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Parallelize the ingestion of ObjectStore sources
3 participants