List optional cytoolz dependency

[jakirkham]

Would be nice if somehow cytoolz was noted as an optional dependency in setup.py. Not totally sure the right place to add it though.

[mrocklin]

Any update here @jakirkham ? It looks like this is stale. Should we close it?

[dhirschfeld]

As someone responsible for specc'ing up and deploying an analytics environment I think this is a pretty important issue. If cytoolz isn't in your environment your code will just run slower... and you'll never know that it could be faster

Th problem for me is discoverability - the default install isn't the fastest and it's hard for a non-developer to know what they need to install if they want the best performance.

That said, it doesn't affect me since I use conda and both AnacondaRecipes and conda-forge have a hard-dependency on cytoolz.

If cytoolz isn't a hard dependency then what would be very useful is just and optional-requirements.txt file which would make it much more discoverable.

[dhirschfeld]

...or just add a 'perf' section to extra_requires - e.g.

extras_require = {
  'array': ['numpy >= 1.11.0', 'toolz >= 0.7.3'],
  'bag': ['cloudpickle >= 0.2.1', 'toolz >= 0.7.3', 'partd >= 0.3.8'],
  'dataframe': ['numpy >= 1.11.0', 'pandas >= 0.19.0', 'toolz >= 0.7.3',
                'partd >= 0.3.8', 'cloudpickle >= 0.2.1'],
  'distributed': ['distributed >= 1.22'],
  'delayed': ['toolz >= 0.7.3'],
  'perf': ['cytoolz >= 0.7.3'],
}

[dhirschfeld]

Note: this also applies to city/murmur hash... which aren't installed by default with either conda or pip.

dask/dask/hashing.py

Lines 12 to 16 in 334bc7f

	# Timings on a largish array:
	# - CityHash is 2x faster than MurmurHash
	# - xxHash is slightly slower than CityHash
	# - MurmurHash is 8x faster than SHA1
	# - SHA1 is significantly faster than all other hashlib algorithms

dask/dask/array/core.py

Lines 2160 to 2165 in df7db48

	name : str, optional
	The key name to use for the array. Defaults to a hash of ``x``.
	By default, hash uses python's standard sha1. This behaviour can be
	changed by installing cityhash, xxhash or murmurhash. If installed,
	a large-factor speedup can be obtained in the tokenisation step.
	Use ``name=False`` to generate a random name instead of hashing (fast)

[dhirschfeld]

Also, scipy - #3309

From my point of view - I would really like these dependencies listed somewhere, especially where they impact performance.

There seems to be some pushback to having another section in the extra_requires. If that's a non starter then perhaps my suggestion of an optional-requirements.txt file above would be acceptable?

[dhirschfeld]

I would use comments to split the optional-requirements.txt file into sections - those deps which will improve performance and those which are only required to interact with 3rd party libraries.

[dhirschfeld]

This appears to be the approach taken by pandas:
https://github.com/pandas-dev/pandas/blob/master/ci/requirements-optional-conda.txt

...by default install just the minimal deps but list (somewhere) all optional deps, including any version requirements.

[mrocklin]

If someone wants to compile a list of all optional dependencies that would be welcome. FWIW I doubt that including cytoolz over toolz will have a significant effect on runtime in most cases. Most other missing dependencies will inform the user with informative error messages.

…

On Wed, Oct 24, 2018 at 10:11 PM Dave Hirschfeld ***@***.***> wrote: This appears to be the approach taken by pandas: https://github.com/pandas-dev/pandas/blob/master/ci/requirements-optional-conda.txt ...by default install just the minimal deps but list (somewhere) all optional deps, including any version requirements. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2812 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszCEzkVRVLbzN3HU3ojPb3kytPyJtks5uoR3hgaJpZM4QDZHI> .

[jakirkham]

FWIW, @dhirschfeld, had attempted this in PR ( #3456 ). I don't plan to carry it forward near term. So feel free to pick that up if you like. Should just be a matter of addressing a few comments in there.

[O-sama12]

@jakirkham Hey Jakir
How can I contribute in this issue? Tell me what's the issue and how can I fix it?