This repository has been archived by the owner. It is now read-only.

multiple threads to compile assets? #308

Closed
josh-m-sharpe opened this Issue Mar 19, 2012 · 7 comments

Comments

Projects
None yet
6 participants
@josh-m-sharpe

josh-m-sharpe commented Mar 19, 2012

When compiling assets, I generally only see one processor being used, which tells me the whole asset compilation process is single-threaded. Is there a reason why it must be that way?

I imagine using threads could significantly improve compiling times.

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Apr 21, 2012

Any news so far? I would like this possibility too, since compilation may consume alot of time and this way, we could greatly reduce deployment time.

ghost commented Apr 21, 2012

Any news so far? I would like this possibility too, since compilation may consume alot of time and this way, we could greatly reduce deployment time.

@jeremy

This comment has been minimized.

Show comment
Hide comment
@jeremy

jeremy Apr 21, 2012

Nobody's working on it, @robertwachs. Please do investigate!

jeremy commented Apr 21, 2012

Nobody's working on it, @robertwachs. Please do investigate!

@steel

This comment has been minimized.

Show comment
Hide comment
@steel

steel Apr 28, 2012

Contributor

I have a fork that handles multi-threaded manifest.compile()
It's located at https://github.com/steel/sprockets/tree/multi-process-compiler

On an 8 core box running with 8 manifest compiler workers we've dropped asset compilation from 7 minutes (1 core intel box) down to 50 seconds. Not too shabby.

I haven't had time to cleanup the code and issue a pull request yet but it does work.

Contributor

steel commented Apr 28, 2012

I have a fork that handles multi-threaded manifest.compile()
It's located at https://github.com/steel/sprockets/tree/multi-process-compiler

On an 8 core box running with 8 manifest compiler workers we've dropped asset compilation from 7 minutes (1 core intel box) down to 50 seconds. Not too shabby.

I haven't had time to cleanup the code and issue a pull request yet but it does work.

@josh josh closed this May 16, 2012

@jjb

This comment has been minimized.

Show comment
Hide comment
@jjb

jjb commented Jun 3, 2012

@steel awesome!

@ghost

This comment has been minimized.

Show comment
Hide comment
@ghost

ghost Jun 4, 2012

Nice solution!
Is there any way using this with Rails 3.2.2 in a clean way? I tried to bundle it but sadly, ActionPack depends on sprockets ~> 2.1.2 so it wont use this version. Also tried to backport it but the structure of Sprockets changed alot.

If someone has a solution for this, please leave a note. Thanks

Edit Seems like @guilleiguaran took care of that: rails/rails#5409 but its not in 3.2 stable yet.

ghost commented Jun 4, 2012

Nice solution!
Is there any way using this with Rails 3.2.2 in a clean way? I tried to bundle it but sadly, ActionPack depends on sprockets ~> 2.1.2 so it wont use this version. Also tried to backport it but the structure of Sprockets changed alot.

If someone has a solution for this, please leave a note. Thanks

Edit Seems like @guilleiguaran took care of that: rails/rails#5409 but its not in 3.2 stable yet.

@steel

This comment has been minimized.

Show comment
Hide comment
@steel

steel Jan 11, 2013

Contributor

Boy its been a long time but finally dropped the monkey patch into a gem:

https://rubygems.org/gems/sprockets-derailleur

You can now compile your manifest with as many processes as you need and I've included a processor detector as well.

processes = SprocketsDerailleur::number_of_processors rescue 1
manifest = Sprockets::Manifest.new(Application::Sprockets, 'public/assets', processes)

We use this multiple times a day deploying all over the place. Hope this helps other out there.

Contributor

steel commented Jan 11, 2013

Boy its been a long time but finally dropped the monkey patch into a gem:

https://rubygems.org/gems/sprockets-derailleur

You can now compile your manifest with as many processes as you need and I've included a processor detector as well.

processes = SprocketsDerailleur::number_of_processors rescue 1
manifest = Sprockets::Manifest.new(Application::Sprockets, 'public/assets', processes)

We use this multiple times a day deploying all over the place. Hope this helps other out there.

@chamnap

This comment has been minimized.

Show comment
Hide comment
@chamnap

chamnap commented Nov 24, 2014

@steel awesome!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.