-
Notifications
You must be signed in to change notification settings - Fork 487
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature][pydevlake] Better support for incremental collection #4880
Comments
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. |
This issue has been automatically marked as stale because it has not had recent activity for 30 days. It will be closed in next 7 days if no further activity occurs. |
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. |
This issue has been closed because it has not received response for too long time. You could reopen it if you encountered similar problems in the future. |
This issue has been automatically marked as stale because it has been inactive for 60 days. It will be closed in next 7 days if no further activity occurs. |
This issue has been closed because it has been inactive for a long time. You can reopen it if you encounter the similar problem in the future. |
Search before asking
Use case
As a python plugin developer, I want to have an easier way of declaring which attribute are used as checkpoint for incremental collection.
Description
Currently,
collect
methods get access to a state dict parameter that they can fill with arbitrary data that they will need in subsequent execution to implement incremental data collection.This is too low-level and complexifies the implementation of those methods.
We should introduce a higher-level "bookmark" or "checkpoint" concept that is used to extract and compare specific JSON properties in order to limit collection to new data only.
Related issues
No response
Are you willing to submit a PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: