Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[admin] add support for re-encoding video Previews #16

Closed
1 task done
eins78 opened this issue Dec 19, 2016 · 4 comments
Closed
1 task done

[admin] add support for re-encoding video Previews #16

eins78 opened this issue Dec 19, 2016 · 4 comments
Assignees
Milestone

Comments

@eins78
Copy link
Contributor

eins78 commented Dec 19, 2016

building on #15, adding support for video is much simpler:
generalize the work already done for audio and adapt it to use the new profile attribute to match previews to their configurations.

Also, we'll need a simple fix to deal with the zencoder rate limit.


tasks:

  • Admin UI: re-encoding "wizard" @eins78

implementation controller logic @michalpodlecki
(details will be written down after making the prototype, will be in form of mock methods)


some docs wtf we are doing and why

Situation:

We need to trigger many jobs from the server.
This takes a long time, we can't make it faster;
but even if we could we are slowed down by the API rate limit

Problem:

If we send 1 request from the browser, in a simple approach
the controller is blocking the response, waiting for the complete queue to finish.
In practice the request will time out, leaving the whole system in an
inconsistent state, because the queue started in the controller is not reliably aborted.

Solution:

The request only deals with the first 100 items in the queue
and responds with the number of remaining items.
A simple JS client takes care of continously sending requests
as long as there are items in the queue.

Prior Art:

  • WordPress Multisite Admin Mass Site Upgrade:
    simpler solution using redirects but much worse UX and hard to makes stable/resiliant.
  • Doku wiki Admin UI Mass Renaming of Pages (very similar solution)

Bonus:

By using a JS client the process could be made more resiliant:

  • While process is running, it is warned against leaving/closing the window
  • there is a "Abort" button which waits for the current request to finish cleanly
  • waiting between requests (for rate limiting)

Deals with rate limits imposed by Zencoder.
The server can just query the DB to check how many ZencoderJobs were sent in the last time and if we should slow down.
If this is the case, the client will receive a 420 status
and wait for 1 minute before trying further (to not overload our own server).

Global Lock:
Only 1 User can open this View in 1 Browser at the time, lock times out after 5 minutes.
(Implemented by storing the session id in special Rails.cache key)

@eins78
Copy link
Contributor Author

eins78 commented Dec 19, 2016

@michalpodlecki Idea: just send the first batch? (100, or maybe a better value than can be found by looking at their docs again)?

We already assume that we won't be able to send all missing previews in one go because
a) the controller/proxy/browser will time out at some point
b) we send to fast and hit the rate limit, so end of the batch will result in failures.

This would also solve the problem of "concurrent batches", the button is clicked and there will a response in reasonable time.
Everything would cleanly be persisted to the DB, and the button could be immediately clicked again.
In this szenario it wouldn't even be too hard to really check for the rate limit - we can just query zencoder_jobs to find out how much we've sent.

@eins78
Copy link
Contributor Author

eins78 commented Dec 20, 2016

@michalpodlecki the JS client is working with a mock controller, you've already seen it ;)
Madek/madek_admin-webapp@35d4fa3

screen shot 2016-12-20 at 20 05 03

screen shot 2016-12-20 at 19 30 30

@eins78
Copy link
Contributor Author

eins78 commented Dec 21, 2016

@eins78
Copy link
Contributor Author

eins78 commented Feb 13, 2017

see #20

@eins78 eins78 closed this as completed Feb 13, 2017
@eins78 eins78 added this to the ~old~ milestone Feb 18, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants