Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Data duplicate detector feature. #1000

Merged
merged 3 commits into from Dec 1, 2022

Conversation

devaskim
Copy link
Contributor

@devaskim devaskim commented Nov 17, 2022

Discussion #986

  1. Duplicate detection is performed after connector has pushed new converted data to gateway 's core module. So detector doesn't depend on any connector specificity.
  2. By default, duplicate detection is disabled.
  3. Each time new converted data is received by gateway's core, detector checks on sendDataOnlyOnChange option in that data.
  4. If the option is present and has false value, data is not checked on duplicates.
  5. If the option is present and has true value, data is checked on duplicates.
  6. If the option is absent, detector asks the connector which has pushed data if duplicate detection is enabled at all (*) or for the device - the origin of data.

(*) In scope of this work ONLY MQTT connector is a bit improved to inform if duplicate detection is enabled for this connector. Other connectors will respond that duplicate detection is disabled for them. Thus users of other connector should either set sendDataOnlyOnChange to true in each converted data (in other words, tune theirs converters) or contribute something similar that was done for MQTT connector in this PR.

thingsboard_gateway/gateway/tb_gateway_service.py Outdated Show resolved Hide resolved
thingsboard_gateway/connectors/mqtt/mqtt_connector.py Outdated Show resolved Hide resolved
thingsboard_gateway/gateway/duplicate_detector.py Outdated Show resolved Hide resolved
thingsboard_gateway/gateway/duplicate_detector.py Outdated Show resolved Hide resolved
thingsboard_gateway/gateway/duplicate_detector.py Outdated Show resolved Hide resolved
thingsboard_gateway/gateway/duplicate_detector.py Outdated Show resolved Hide resolved
@samson0v
Copy link
Contributor

@devaskim, I will test your changes

@samson0v samson0v merged commit cca2d24 into thingsboard:master Dec 1, 2022
@devaskim
Copy link
Contributor Author

devaskim commented Dec 1, 2022

@samson0v what about previous PR for sendDataOnlyOnChange? Will it be reverted?

@samson0v
Copy link
Contributor

samson0v commented Dec 7, 2022

@devaskim not for now. We will see the result of using Duplicate Service.

@devaskim
Copy link
Contributor Author

devaskim commented Dec 7, 2022

@samson0v Ok, I just want to ensure that this item is on tracking.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants