Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multiprocessing in MLE model prevents use on AWS Lambda Functions #79

Closed
tch521 opened this issue Mar 5, 2024 · 1 comment
Closed
Assignees
Labels
enhancement New feature or request

Comments

@tch521
Copy link

tch521 commented Mar 5, 2024

(Firstly I'll just say that this is a fantastic library and very easy to use, great job!)

For whatever reason, AWS lambda functions cannot use multiprocessing.Pool(). This is a problem as it is used by pyextremes when fitting an MLE model. So when I attempt to run my AWS lambda function, it crashes, and the current solution is for me to modify line 199 in model_mle.py to if True, thereby ensuring it does not use the multiprocessing approach.

This is not ideal. It's slightly tedious, but AWS describes this method for using multiprocessing that is compatible with lambda functions. (Instead of using Pool you use Process and Pipe with some additional code to set it up.) Alternatively you could just add a kwarg to avoid the multiprocessing method which would be fine too for those of us not concerned with speed, and much easier to implement.

@tch521 tch521 added the enhancement New feature or request label Mar 5, 2024
@georgebv
Copy link
Owner

georgebv commented Mar 5, 2024

Adding an option like use_multiprocessing which is True by default sounds like a good solution. If you open a PR I'll gladly review and merge.

@georgebv georgebv closed this as completed May 3, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants