Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Measure the volume of queries from bifrost gateways #93

Open
masih opened this issue Apr 13, 2023 · 2 comments
Open

Measure the volume of queries from bifrost gateways #93

masih opened this issue Apr 13, 2023 · 2 comments

Comments

@masih
Copy link
Member

masih commented Apr 13, 2023

To truly measure this, we would have to enable event logs on cloudfront which would log every request flushed to a S3 bucket on hourly basis. Then write code to process it, filtering requests by the gateway IPs.
@masih
Copy link
Member Author

masih commented Apr 13, 2023

For quicker sanity check, write a metric in indexstar that checks the source IP against a well known list of gateway IPs

masih added a commit to ipni/storetheindex that referenced this issue Apr 23, 2023
@masih
Enable CloudFront logging in prod
18e5df7 
Enable event logs in prod CloudFront to:

1. check gateway traffic is reaching us.

2. collect data for a research paper in the making by probelab.

Relates to:
 - ipni/indexstar#93
@masih masih mentioned this issue Apr 23, 2023
Merged
masih added a commit to ipni/storetheindex that referenced this issue Apr 23, 2023
@masih
Enable CloudFront logging in prod
ecc2996 
Enable event logs in prod CloudFront to:

1. check gateway traffic is reaching us.

2. collect data for a research paper in the making by probelab.

Relates to:
 - ipni/indexstar#93
@masih
Copy link
Member Author

masih commented Apr 25, 2023

Re-run the analysis on 2 days (ish) worth of event logs and none of the source IPs match the 120 IPFS gateway IP addresses.

In the meantime I have disabled event log collection on cid.contact cloudfront cache to reduce excessive log collection.
@gmasgras @guseggert please give me a shout whenever you'd like me to re-enable this and run the IP check. 🙏

The tool used to analyse logs can be found here. Please let me know if you need access to the raw event logs.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@masih