New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parallel memory warnings #208

Merged
merged 7 commits into from Aug 13, 2018

Conversation

Projects
None yet
3 participants
@rwedge
Contributor

rwedge commented Aug 10, 2018

There are two main parts to this pull request

  • Change to how Featuretools sets worker memory limit by default: previously, reducing the number of distributed workers would not increase the maximum memory allotted to each worker. With this change, Featuretools will set memory limits by splitting the total memory on the system equally between workers instead of equally between total cores on the system.
  • Raise errors or warnings about worker memory limits: Featuretools will now raise an error if it is about to transmit the EntitySet to a worker and realizes the worker's memory limit is smaller than the size of the EntitySet. Featuretools will also raise a warning if the worker's memory limit is less than twice the size of the EntitySet.
@codecov-io

This comment has been minimized.

codecov-io commented Aug 10, 2018

Codecov Report

Merging #208 into master will increase coverage by 0.03%.
The diff coverage is 96.55%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #208      +/-   ##
==========================================
+ Coverage   93.36%   93.39%   +0.03%     
==========================================
  Files          70       71       +1     
  Lines        7744     7785      +41     
==========================================
+ Hits         7230     7271      +41     
  Misses        514      514
Impacted Files Coverage Δ
...computational_backends/calculate_feature_matrix.py 96.98% <ø> (ø) ⬆️
featuretools/tests/testing_utils/__init__.py 100% <100%> (ø) ⬆️
featuretools/tests/testing_utils/cluster.py 100% <100%> (ø)
featuretools/computational_backends/utils.py 94.9% <100%> (+0.53%) ⬆️
...utational_backend/test_calculate_feature_matrix.py 99.28% <93.93%> (+0.02%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d1ab519...8f43aa4. Read the comment docs.

@kmax12

This comment has been minimized.

Member

kmax12 commented Aug 10, 2018

Looks good, but let's add a test case for a user supplied memory limit

@kmax12

This comment has been minimized.

Member

kmax12 commented Aug 13, 2018

Looks good to me. Merging

@kmax12 kmax12 merged commit 6c756f5 into master Aug 13, 2018

2 checks passed

ci/circleci Your tests passed on CircleCI!
Details
license/cla Contributor License Agreement is signed.
Details

@kmax12 kmax12 deleted the parallel_memory_warnings branch Oct 2, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment