Purge Database Tables

[blueandgold]

Currently, the inventory database tables are not purged. So, they stay in place, and grows over time. Which makes it harder to navigate and exceeds the database's capacity. So, we need to provide a way for users to purge these tables. Possibly by a config value that users can specify.

[gbrindisi]

This is a problem we would like to solve as well :) I'm commenting to remember to get back with ideas later

[forseti-security-waffle-bot]

Issue review: This is something the team can't get to in the near-term.

[victorvess]

@blueandgold I am interested in picking this up - my thought is to provide a retention_days property in the inventory section of forseti config, that will cause inventory scan to clean up any snapshot tables older than that time period (basically delete tables like '%_cycle_timestamp')

Is there any other dimension than age in days that should be considered?

Also - do we think the corresponding rows within snapshot_cycles should be cleaned up too? I am not sure if there is any kind of audit benefit of leaving those around, I was leaning toward leaving them just in case because it shouldn't take up much space (<9000 rows per year).

This would be my first contribution to Forseti, let me know if you need any additional artifacts

[blueandgold]

Hi Victor, thank you for your input here! Yes, I think that your approach to use a retention_days is more flexible and comprehensive, and will also cover angelo's use case as well. Please make sure to default it to a state where no data will be purged (perhaps with -1).

Keeping the snapshot data makes sense to me, and preferable as it might be helpful for people to have a sense of when forseti has run previously. As you pointed out, this is very inexpensive to do as well.

Looking forward to your PR, and please ask any questions!

[blueandgold]

This also looks good to @angelomellos in PR #1103 .

[victorvess]

Already submitted #1129 - per your comment around defaulting, I do this if the directive is missing, but I put a non-zero default in the forseti_conf.yaml.in - would you prefer if I set that to 0 (keep always) instead of 30 days?

[blueandgold]

Fixed by #1129