Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Switch to new cache if it processed enough number of operations #1092

Merged
merged 1 commit into from
Mar 1, 2023
Merged

feat: Switch to new cache if it processed enough number of operations #1092

merged 1 commit into from
Mar 1, 2023

Conversation

chubei
Copy link
Contributor

@chubei chubei commented Feb 28, 2023

This PR also solves the switching problem.

@chubei chubei requested a review from v3g42 February 28, 2023 10:17
@chubei chubei linked an issue Feb 28, 2023 that may be closed by this pull request
gRPC query returns empty records when there is data in teh cahce. #1089
Closed
@chubei chubei changed the title feat: Switch to new cache if it processed enought number of operations feat: Switch to new cache if it processed enough number of operations Feb 28, 2023
@coveralls
Copy link

Pull Request Test Coverage Report for Build 4302644661

  • 6 of 39 (15.38%) changed or added relevant lines in 1 file are covered.
  • 95 unchanged lines in 3 files lost coverage.
  • Overall coverage increased (+0.3%) to 71.429%
Changes Missing Coverage Covered Lines Changed/Added Lines %
dozer-orchestrator/src/pipeline/sinks.rs 6 39 15.38%
Files with Coverage Reduction New Missed Lines %
dozer-ingestion/src/connectors/object_store/table_reader.rs 1 90.38%
dozer-orchestrator/src/pipeline/sinks.rs 37 65.43%
dozer-core/src/record_store.rs 57 82.78%
Totals Coverage Status
Change from base Build 4302087115: 0.3%
Covered Lines: 28515
Relevant Lines: 39921
💛 - Coveralls

@chubei chubei merged commit 6a05167 into getdozer:main Mar 1, 2023
@chubei chubei deleted the feat/count branch March 1, 2023 11:19
mediuminvader added a commit that referenced this pull request Mar 8, 2023
@karolisg @chubei
@readall @mediuminvader @snork-alt
refactor: Use rdkafka instead of kafka library (#1132)
713e2a2 
* chore: Move `SnapshottingDone` out of `Operation` so processors don't have to know it.(#1103)

Also removed unused variants of `IngestionMessage`

* feat: Switch to new cache if it processed enought number of operations (#1092)

* chore: Don't create default eth provider (#1108)

* chore: Remove an unused `Arc<RwLock>` (#1106)

* fix: updated download_binary (#1107)

* fix: updated download_binary

* fix: updated download_binary

* fix: updated download_binary

* fix: updated download_binary

* fix: updated download_binary

* chore: Provide context to `ApiError`s (#1109)

* tests: Run postgres tests on github actions and add snowflake support (#1037)

* chore: Add installation to and remove docker instructions from README (#1110)

* chore: Add remove unnecessary libraries remove (#1113)

* chore: Don't run coverage on `pull_request_target` (#1116)

* refactor: Simplify schemas fetch query (#1114)

* refactor: Simplify schemas fetch query

Fixes #1073

* Fix formatting

* Fix tests

* chore: Use 16 cores runner for coverage and release, 4 cores runner for lint (#1118)

* chore: Use correct proto include path to avoid rebuilding every time (#1115)

* fix: Sink should only be built after all source checkpoints are checked (#1112)

* chore: Use `Dag::map_owned` instaed of `Dag::map`

* chore: Remove context from `DagSchemas::EdgeType`

* fix: Sink should only be built after all source checkpoints are checked

* chore: Fix docker image publishing steps in release workflow (#1120)

* remove unused dependency (#1122)

The unused dependency of unixstring was causing build failure of native windows binary.

Signed-off-by: Shailendra Thakur <31568561+readall@users.noreply.github.com>

* fix: Blue cache was counted with limit. Green cache didn't get picked up. (#1123)

* fix: Define unique temp folder for aggregation tests (#1124)

* fix: Define unique tmp dir

* fix clippy

* fix: groupby without measures were handled wrongly (#1127)

* fix groupby withou measures

* clippy fix

* chore: Remove some unnecessary `clone`s (#1131)

* chore: Don't clone whole config around the orchestrator

* chore: Remove exccesive cloneing in `SourceBuilder`

* chore: Remove excessive cloning in `PipelineBuilder`

* chore: Remove excessive cloning in `Executor`

* chore: clippy fix

* refactor: Use rdkafka instead of kafka library

* Format

---------

Signed-off-by: Shailendra Thakur <31568561+readall@users.noreply.github.com>
Signed-off-by: Dario Pizzamiglio <1150680+mediuminvader@users.noreply.github.com>
Co-authored-by: Bei Chu <914745487@qq.com>
Co-authored-by: Chloe Kim <chloe@getdozer.io>
Co-authored-by: Shailendra Thakur <31568561+readall@users.noreply.github.com>
Co-authored-by: Dario Pizzamiglio <1150680+mediuminvader@users.noreply.github.com>
Co-authored-by: Matteo Pelati <matteo.pelati@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@v3g42 v3g42 v3g42 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

gRPC query returns empty records when there is data in teh cahce.
3 participants
@chubei @coveralls @v3g42