New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for background jobs? #167

Open
lserman opened this Issue Feb 28, 2015 · 5 comments

Comments

Projects
None yet
4 participants
@lserman

lserman commented Feb 28, 2015

Uploading directly to s3 is great, but the copy from the cache to store is still done inline (even the s3 copy command can be pretty slow). I'm trying to find the best way to background this process with ActiveJob so that the cache -> store transfer can be deferred while the image is stored from the cache.

My original idea was to make a backend that wraps another backend, deferring all methods to the wrapped backend except for upload/delete, which it would override to enqueue the worker.

The problem is that most of the work (uploading cache -> store, deleting from cache, updating model) is done by the attacher, not by the backend. Also, the backend doesn't really have any context on what record it is working for, so there is no way to enqueue anything that will know what to do after popping out of Redis.

Any ideas on how this could done? I don't mind making a PR but I'm not really sure where to begin.

@jnicklas

This comment has been minimized.

Show comment
Hide comment
@jnicklas

jnicklas Mar 1, 2015

Contributor

The simple way of doing this would be just disabling the before hook somehow. There currently isn't an option to do this, so it might not be super easy. Maybe we should facilitate this somehow.

Contributor

jnicklas commented Mar 1, 2015

The simple way of doing this would be just disabling the before hook somehow. There currently isn't an option to do this, so it might not be super easy. Maybe we should facilitate this somehow.

@lserman

This comment has been minimized.

Show comment
Hide comment
@lserman

lserman Mar 2, 2015

Hmm... correct me if I'm wrong, but if the before_save is disabled, the attachment ID field won't make it past the save since that field is set in Refile::Attacher#store!. I think ideally the attachment ID field would be set regardless of if store! is called, that way it can still be accessed through the cache until the background worker has finished.

If we could specify an adapter for the attacher as well as the backend, then this might be a little easier? Then there could be a subclass of Refile::Attacher that overrides store! with the background functionality.

lserman commented Mar 2, 2015

Hmm... correct me if I'm wrong, but if the before_save is disabled, the attachment ID field won't make it past the save since that field is set in Refile::Attacher#store!. I think ideally the attachment ID field would be set regardless of if store! is called, that way it can still be accessed through the cache until the background worker has finished.

If we could specify an adapter for the attacher as well as the backend, then this might be a little easier? Then there could be a subclass of Refile::Attacher that overrides store! with the background functionality.

@dhyegofernando

This comment has been minimized.

Show comment
Hide comment
@dhyegofernando

dhyegofernando Apr 20, 2015

Specially thinking about non CPU blocker design for requests (e.g. APIs/webservices), this could be pretty useful.

👍 for it

dhyegofernando commented Apr 20, 2015

Specially thinking about non CPU blocker design for requests (e.g. APIs/webservices), this could be pretty useful.

👍 for it

@mskog

This comment has been minimized.

Show comment
Hide comment
@mskog

mskog May 12, 2015

👍 seems very useful

There is something in https://github.com/lserman/refile-backgrounder but I haven't tried it yet

mskog commented May 12, 2015

👍 seems very useful

There is something in https://github.com/lserman/refile-backgrounder but I haven't tried it yet

@lserman

This comment has been minimized.

Show comment
Hide comment
@lserman

lserman May 12, 2015

Hi @mskog that is my gem but it is certainly not ready for production yet. Mostly I just have it there for experimentation, you are welcome to fork or contribute though. I am using it in a side project and haven't had any major issues so far. I would like to add tests to it of course... we all know how it goes.

lserman commented May 12, 2015

Hi @mskog that is my gem but it is certainly not ready for production yet. Mostly I just have it there for experimentation, you are welcome to fork or contribute though. I am using it in a side project and haven't had any major issues so far. I would like to add tests to it of course... we all know how it goes.

@jnicklas jnicklas modified the milestone: v1.0 May 17, 2015

@jnicklas jnicklas modified the milestones: v1.0, v1.1 May 29, 2015

@jnicklas jnicklas modified the milestones: v1.1, v1.0 Sep 9, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment