Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingest check for duplicate entries #439

Merged
merged 7 commits into from
Jul 7, 2023

Conversation

sandreae
Copy link
Member

@sandreae sandreae commented Jul 7, 2023

Check if an entry already exists on the node before consuming in Ingest. If the entry is present, don't ingest it again and don't prematurely close the sync session.

This is to account for the case where the same entry is sent by two different peers at the same time.

📋 Checklist

  • Add tests that cover your changes
  • Add this PR to the Unreleased section in CHANGELOG.md
  • Link this PR to any issues it closes
  • New files contain a SPDX license header

@sandreae sandreae linked an issue Jul 7, 2023 that may be closed by this pull request
@codecov
Copy link

codecov bot commented Jul 7, 2023

Codecov Report

Patch coverage: 93.10% and project coverage change: +0.03 🎉

Comparison is base (ae27ff2) 90.05% compared to head (bdf9d48) 90.09%.

❗ Current head bdf9d48 differs from pull request most recent head cab960f. Consider uploading reports for the commit cab960f to get more accurate results

Additional details and impacted files
@@               Coverage Diff               @@
##           development     #439      +/-   ##
===============================================
+ Coverage        90.05%   90.09%   +0.03%     
===============================================
  Files               87       87              
  Lines             8439     8461      +22     
===============================================
+ Hits              7600     7623      +23     
+ Misses             839      838       -1     
Impacted Files Coverage Δ
aquadoggo/src/replication/manager.rs 88.95% <75.00%> (-0.29%) ⬇️
aquadoggo/src/replication/ingest.rs 91.11% <100.00%> (+6.17%) ⬆️

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

@sandreae sandreae marked this pull request as ready for review July 7, 2023 07:50
@adzialocha adzialocha merged commit 3812076 into development Jul 7, 2023
5 of 6 checks passed
@adzialocha adzialocha deleted the ingest-check-for-duplicate-entries branch July 7, 2023 11:29
adzialocha added a commit that referenced this pull request Jul 14, 2023
* development: (23 commits)
  Implement `dialer` behaviour (#444)
  Sort expected results in strategy tests
  Update CHANGELOG
  Replicate operations in topo order (#442)
  Maintain sorted operation indexes (#438)
  Use fork of `asynchronous-codec`  (#440)
  Ingest check for duplicate entries (#439)
  Reverse lookup for pinned relations in dependency task (#434)
  Remove unnecessary exact version pinning in Cargo.toml
  Make `TaskInput` an enum and other minor clean ups in materialiser (#429)
  Use `libp2p` `v0.52.0` (#425)
  Fix race condition when check for existing view ids was too early (#420)
  Reduce logging verbosity
  CI: Temporary workaround for Rust compiler bug (#417)
  Fix early document view insertion (#413)
  Handle duplicate document view insertions (#410)
  Decouple p2panda's authentication data types from libp2p's (#408)
  Remove dead_code attribute in lib
  Integrate replication manager with networking stack (#387)
  Implement naive replication protocol (#380)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ignore duplicate data arriving at the ingest
2 participants