New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to Age-Out (Purge Old) Data? #45

Closed
nickwallen opened this Issue Apr 25, 2016 · 5 comments

Comments

Projects
None yet
3 participants
@nickwallen
Copy link

nickwallen commented Apr 25, 2016

Is there a mechanism to purge or age-out old data with OpenTAXII? How do I ensure that the data store doesn't grow without bounds?

@traut

This comment has been minimized.

Copy link
Member

traut commented Apr 25, 2016

@nickwallen default SQL backend does not support any clean-up procedures at the moment.

What would be your requirements here?

@nickwallen

This comment has been minimized.

Copy link

nickwallen commented Apr 25, 2016

I am looking at integrating OpenTAXII as a source of threat intel data for the default deployment of Apache Metron. In most cases, I envision Metron users leveraging their own existing aggregator platform that would be used instead. Although if none exists, I am hoping that OpenTAXII can fill that role.

I have some logic that will likely be added to Metron that deploys OpenTAXII and can sync with some external 3rd party feeds. If I setup a mechanism that continually syncs external threat intel feeds, I need to ensure that the data store size doesn't grow without bound. That's what I need.

@code040

This comment has been minimized.

Copy link
Contributor

code040 commented Apr 25, 2016

Are you planning of running this using the SQLite database? Then all you can do is implement a cleanup routine inside the persistence layer (and possible in the manager), which runs periodically any cleanup task. As @traut mentioned, this is currently not the case.

@traut

This comment has been minimized.

Copy link
Member

traut commented Apr 25, 2016

@nickwallen another option would be to implement another OpenTAXII CLI command to clean-up DB. Something as simple as "delete content blocks older than X in collections A and B". Host admin can use this inside scheduled cron jobs // @code040

@nickwallen

This comment has been minimized.

Copy link

nickwallen commented Apr 25, 2016

Yes, I like that idea. Ideally, it would work the same no matter what persistence API (sqlite, mysql, etc) is used.

traut added a commit that referenced this issue Jun 1, 2016

@traut traut closed this Jun 1, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment