Add Fastly log processor #1176
Merged
Add Fastly log processor #1176
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
1 task
|
Rubocop is not a happy. 😱 I'll amend the commits. 😄 |
ktheory
force-pushed
the
fastly-log-processor
branch
2 times, most recently
from
4158ccb
to
13f1bab
Compare
| @@ -3,4 +3,4 @@ | |||
| Delayed::Worker.max_run_time = 5.minutes | |||
| Delayed::Worker.logger = Logger.new(Rails.root.join('log/delayed_job.log')) | |||
|
|
|||
| PRIORITIES = { push: 1, download: 2, web_hook: 3 }.freeze | |||
| PRIORITIES = { push: 1, download: 2, web_hook: 3, download_metrics: 4 }.freeze | |||
There was a problem hiding this comment.
Can we shorten this to stats?
There was a problem hiding this comment.
I hate this constant been here :( eventually i wanna moved it out. I had that somewhere. Anyways, not in the scope of this PR
|
I added a few comments and it looks like this needs a rebase. |
|
|
||
| # TODO: set real queue name | ||
| # TODO: set auto_delete: true after testing | ||
| shoryuken_options queue: 'TODO-add-real-queue', body_parser: :json, auto_delete: false |
There was a problem hiding this comment.
Do we actually need Shoryuken here? What's the advantage to using this when we have Delayed::Job?
There was a problem hiding this comment.
Do we actually need Shoryuken here? What's the advantage to using this when we have Delayed::Job?
We need a tool that uses AWS SQS as the queue. See this comment & discussion on rubygems-infrastructure.
ktheory
force-pushed
the
fastly-log-processor
branch
2 times, most recently
from
b100d0c
to
15de30b
Compare
ktheory
force-pushed
the
fastly-log-processor
branch
from
f9b3c19
to
1fcebe6
Compare
We’ll use shoryuken for processing SQS messages; and aws-sdk for downloading S3 files
It takes an S3 bucket & key, parses out the download counts, and bulk updated download counts in redis
The Shoryuken worker reads S3 ObjectCreated messages from SQS, and enqueues an associated FastlyLogProcessor job.
- count 304 responses as download attempts - rename download_metrics queue to stats Also add a test that 404 responses are not counted as downloads.
Now it relies on an SQS_QUEUE env var
ktheory
force-pushed
the
fastly-log-processor
branch
from
1fcebe6
to
786c20b
Compare
|
LGTM 🚢 |
ktheory
changed the title
1 task
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
👋 Happy Friday 🙆♀️
This implements much of rubygems/rubygems-infrastructure#35, allowing us to update Rubygem download counts in bulk using Fastly access logs rather than on each HTTP request.
I still have some cleanup to do, but wanted to get some 👀 earlier.
To Do
stat-updateC library as part of the feature flagshoryukenprocess on deploy once Chef provisions a runit script for itDeploy plan
NB that this code is disabled right now. We're not currently running a
shoryukenworker, and so nothing would enqueue aFastlyLogProcessorjob.Once this is merged, the next step is to start running a
shoryukenprocess, withFastlyLogProcessoronly logging results & not updating redis.After that I'll toggle the feature flag so
FastlyLogProcessorupdates stats, and disablestat-updateper request.FYI @dwradcliffe