use process pool to compile faster #2377
Conversation
reflex/app.py
Outdated
| @@ -82,6 +84,118 @@ def default_overlay_component() -> Component: | |||
| return connection_modal() | |||
|
|
|||
|
|
|||
| class ExecutorSafeFunctions: | |||
There was a problem hiding this comment.
feels like this class belongs in reflex/compiler/utils.py
There was a problem hiding this comment.
Good idea. IMO the gymnastics involved are little icky so encapsulating as much as we can would be good. More to do in future on that front.
reflex/app.py
Outdated
| @staticmethod | ||
| def compile_page(route: str): | ||
| """Compile a page. | ||
|
|
||
| Args: | ||
| route: The route of the page to compile. | ||
|
|
||
| Returns: | ||
| The path and code of the compiled page. | ||
| """ | ||
| return compiler.compile_page( | ||
| *ExecutorSafeFunctions.COMPILE_PAGE_ARGS_BY_ROUTE[route] | ||
| ) |
There was a problem hiding this comment.
is there a particular reason to go for the staticmethod over classmethod when these do in fact access members of the class?
seems like that would have less code duplication and make it easier to subclass/monkeypatch if some behavior modifications were needed downstream.
There was a problem hiding this comment.
No particular reason. These things started out as full global vars or functions, in fact! The key property is only that the forked child process is able to get at the data / functions correctly. I can try to see if @classmethod works as well.
* Revert "Revert "Revert "Revert "use process pool to compile faster (#2377)" (#2434)" (#2497)" (#2595)" This reverts commit 6b6eea4. * Adjust number of operations for more correct progress bar * app: recognize REFLEX_COMPILE_PROCESSES and REFLEX_COMPILE_THREADS Control whether multiprocessing is used and the number of processes or threads that should be used. This will allow users to opt-in to the new, potentially hazardous, multiprocessing mode, which results in much faster compiles, but has already been reverted 4 times. Lets leave the code in this time, but use the thread pool executor by default. Limiting the number of threads or processes to 1 can also aid in debugging issues that arise during compile time. * Allow REFLEX_COMPILE_PROCESSES=0 to trigger multiprocessing with auto workers
We also do some general cleanup - for steps that are not being parallelized, it should run outside of the executor context manager.
As an aside, we stop using multi-threading to write out final compilation output to file system. Even for a large project like reflex-web, that part takes about 200ms on my laptop (Macs are fancy, but pretty much everyone has SSDs).
Before (reflex-web):
After (reflex-web):
