Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set openstreetmap-fastly-logs S3 retention to 30 days #727

Closed
pnorman opened this issue Aug 6, 2022 · 6 comments
Closed

Set openstreetmap-fastly-logs S3 retention to 30 days #727

pnorman opened this issue Aug 6, 2022 · 6 comments
Labels
location:aws Services hosted on AWS

Comments

@pnorman
Copy link
Collaborator

pnorman commented Aug 6, 2022

In #698 I suggested changing the retention to 30 days for the raw fastly logs, as the successful request data logs will contain all the same information for successful tiles, so the only loss is failed requests and non-tile requests, which we've never needed.

Since the log aggregation has been running for a couple weeks without issues, I recommend turning on a bucket policy to delete files >30 days old. This will bring the bucket size down from 86TB to about 4TB.
@Firefishy
Copy link
Member

I have now enabled this, except for now I have used 90 day object expiry.

@Firefishy
Copy link
Member

I will keep ticket open until object expiry confirmed.

@Firefishy
Copy link
Member

Current size: 79TB
Number of Objects: 3.5M

@Firefishy
Copy link
Member

Current size: 18.6 TB
Number of Objects: 847,374

@Firefishy
Copy link
Member

New limits set.

@Firefishy Firefishy changed the title Set openstreetmap-fastly-logs retention to 30 days Set openstreetmap-fastly-logs S3 retention to 30 days Aug 14, 2022
@Firefishy
Copy link
Member

11th August: 79.5 TB
13th August: 4.5 TB

@Firefishy Firefishy added the location:aws Services hosted on AWS label Aug 14, 2022
@pnorman pnorman mentioned this issue Dec 16, 2022
Closed
@Firefishy Firefishy removed their assignment Aug 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
location:aws Services hosted on AWS
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Firefishy @pnorman