You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
building on #15, adding support for video is much simpler:
generalize the work already done for audio and adapt it to use the new profile attribute to match previews to their configurations.
Also, we'll need a simple fix to deal with the zencoder rate limit.
implementation controller logic @michalpodlecki
(details will be written down after making the prototype, will be in form of mock methods)
some docs wtf we are doing and why
Situation:
We need to trigger many jobs from the server.
This takes a long time, we can't make it faster;
but even if we could we are slowed down by the API rate limit
Problem:
If we send 1 request from the browser, in a simple approach
the controller is blocking the response, waiting for the complete queue to finish.
In practice the request will time out, leaving the whole system in an
inconsistent state, because the queue started in the controller is not reliably aborted.
Solution:
The request only deals with the first 100 items in the queue
and responds with the number of remaining items.
A simple JS client takes care of continously sending requests
as long as there are items in the queue.
Prior Art:
WordPress Multisite Admin Mass Site Upgrade:
simpler solution using redirects but much worse UX and hard to makes stable/resiliant.
Doku wiki Admin UI Mass Renaming of Pages (very similar solution)
Bonus:
By using a JS client the process could be made more resiliant:
While process is running, it is warned against leaving/closing the window
there is a "Abort" button which waits for the current request to finish cleanly
waiting between requests (for rate limiting)
Deals with rate limits imposed by Zencoder.
The server can just query the DB to check how many ZencoderJobs were sent in the last time and if we should slow down.
If this is the case, the client will receive a 420 status
and wait for 1 minute before trying further (to not overload our own server).
Global Lock:
Only 1 User can open this View in 1 Browser at the time, lock times out after 5 minutes.
(Implemented by storing the session id in special Rails.cache key)
The text was updated successfully, but these errors were encountered:
@michalpodlecki Idea: just send the first batch? (100, or maybe a better value than can be found by looking at their docs again)?
We already assume that we won't be able to send all missing previews in one go because
a) the controller/proxy/browser will time out at some point
b) we send to fast and hit the rate limit, so end of the batch will result in failures.
This would also solve the problem of "concurrent batches", the button is clicked and there will a response in reasonable time.
Everything would cleanly be persisted to the DB, and the button could be immediately clicked again.
In this szenario it wouldn't even be too hard to really check for the rate limit - we can just query zencoder_jobs to find out how much we've sent.
building on #15, adding support for video is much simpler:
generalize the work already done for audio and adapt it to use the new
profile
attribute to match previews to their configurations.Also, we'll need a simple fix to deal with the zencoder rate limit.
tasks:
implementation controller logic @michalpodlecki
(details will be written down after making the prototype, will be in form of mock methods)
some docs wtf we are doing and why
Situation:
We need to trigger many jobs from the server.
This takes a long time, we can't make it faster;
but even if we could we are slowed down by the API rate limit
Problem:
If we send 1 request from the browser, in a simple approach
the controller is blocking the response, waiting for the complete queue to finish.
In practice the request will time out, leaving the whole system in an
inconsistent state, because the queue started in the controller is not reliably aborted.
Solution:
The request only deals with the first 100 items in the queue
and responds with the number of remaining items.
A simple JS client takes care of continously sending requests
as long as there are items in the queue.
Prior Art:
simpler solution using redirects but much worse UX and hard to makes stable/resiliant.
Bonus:
By using a JS client the process could be made more resiliant:
Deals with rate limits imposed by Zencoder.
The server can just query the DB to check how many ZencoderJobs were sent in the last time and if we should slow down.
If this is the case, the client will receive a
420
statusand wait for 1 minute before trying further (to not overload our own server).
Global Lock:
Only 1 User can open this View in 1 Browser at the time, lock times out after 5 minutes.
(Implemented by storing the session id in special Rails.cache key)
The text was updated successfully, but these errors were encountered: