Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure densification. #210

Merged
merged 4 commits into from Nov 19, 2018
Merged

Conversation

hameerabbasi
Copy link
Collaborator

Add an environment variable to allow configuration of densification.

@codecov
Copy link

codecov bot commented Nov 19, 2018

Codecov Report

Merging #210 into master will increase coverage by <.01%.
The diff coverage is 100%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #210      +/-   ##
==========================================
+ Coverage   97.66%   97.67%   +<.01%     
==========================================
  Files          11       11              
  Lines        1502     1505       +3     
==========================================
+ Hits         1467     1470       +3     
  Misses         35       35
Impacted Files Coverage Δ
sparse/coo/core.py 97.17% <100%> (+0.02%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6e4d35e...80aeb17. Read the comment docs.

@hameerabbasi hameerabbasi merged commit 0b2cc43 into pydata:master Nov 19, 2018
import os

_AUTO_DENSIFICATION_ENABLED = bool(int(os.environ.get('SPARSE_AUTO_DENSIFY', '1')))

Copy link
Member

@asmeurer asmeurer Nov 19, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would also add

def set_auto_densify(flag):
    """
    Enable or disable automatic densification

    By default it is enabled, unless the environment variable ``SPARSE_AUTO_DENSIFY`` is ``0``.
    
    If automatic densification is disabled, calls like ``np.array(COO)`` will raise ``RuntimeError``.
    """
    global _AUTO_DENSIFICATION_ENABLED
    _AUTO_DENSIFICATION_ENABLED = flag

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem is that that isn't thread safe. For example, if you do it with Dask or whatever else, it can blow stuff up.

An environment variable loaded at import time is guaranteed to be thread safe.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we add a warning to the function that it isn't thread safe. I think if it is only called right after import it should be fine too.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm worried about exactly the use-case you describe, too. :-) Libraries setting this option and users getting confused about what's happening, unfortunately there's no clear way to make it user facing but not library facing.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

unfortunately there's no clear way to make it user facing but not library facing.

That's in the same category as numpy.set_printoptions, np.seterr, private class attributes, etc. - just don't mess with it as a library. It doesn't happen a lot, and if it does the first person who notices files a bug against that library.

@mrocklin
Copy link
Collaborator

mrocklin commented Nov 20, 2018 via email

@hameerabbasi hameerabbasi deleted the configure-densify branch December 5, 2018 15:04
@asmeurer asmeurer mentioned this pull request Dec 12, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants