tlz package: cytoolz or toolz

[eriknw]

Create a package tlz whose modules are populated by cytoolz if available or toolz. This is to provide a convenient way to import the faster of toolz or cytoolz. For example:

>>> import toolz  # must import toolz before importing tlz
>>> from tlz import join  # `cytoolz.join` if cytoolz is installed
>>> import tlz.playground  # from `toolz.playground`, because it's not in `cytoolz`.

This is largely experimental. Something that provides this functionality has been on the radar for a while.

By having tlz be a part of toolz, it will always be available if toolz or cytoolz is installed (i.e., one doesn't need to pip install tlz). A disadvantage of the current approach is that a user needs to import toolz before importing tlz. Is there a good way to make import tlz work without first importing toolz?

Python 2.6 dropped, because it doesn't have importlib, and developing a compatible way to do this would be annoying. Dealing with the import system across Python 2.7 and Python 3.3+ is already annoying enough.

[mrocklin]

Would it be possible to do this without import tricks by having a second top-level package named tlz and including that package in setup.py?

[mrocklin]

For some non-performance-critical functions like pipe we may want to always use toolz for introspection reasons

[eriknw]

Yeah, I expect it's possible to have a second top-level package tlz that we handle in setup.py. I've never done this before though. Do you just call setup twice, or is there a better way to do it?

Since we manually construct new modules for tlz, it should be easy to include a set of exceptions to always pull from toolz. What else should always come from toolz?

[mrocklin]

I think that you create a new top level package directory tlz/ alongside toolz/ and add tlz in the packages field of setup.py. It then installs both. We did this for a while in dask with smaller packages that were co-bundled.

[eriknw]

Would it be possible to do this without import tricks...

Oh, you don't want any import tricks? Would you rather create files for every module? i.e., something like this:

try:
    import cytoolz.curried as mod
except ImportError:
    import toolz.curried as mod
orig_globals = dict(globals())
globals().update(mod.__dict__)
# TODO: handle submodules such as `toolz.curried.operator`
globals().update(orig_globals)
del mod
del orig_globals

I don't think we should be shy about using importlib and sys.meta_path. Once we get this working (and it is), I doubt we'll ever need to touch it again, and it provides a single place to introduce customization logic.

[mrocklin]

OK. Great

[eriknw]

btw, my first attempt did use a separate tlz package with files for every module (I like simplicity too even if it's a little more structured and verbose). There were enough nuances that I ended up making convenience functions that were basically the same as TlzLoader. Since I already had and needed code that could serve as a module finder and loader, it became less burdensome to actually make a module finder and loader. And, truth be told, I found having files for every module to be tiresome.

My plan is to make a separate tlz package as you suggested, and it will dynamically create its modules by adding TlzLoader to sys.meta_path. Technical details aside, do you think this is a good idea, and do you like the name tlz?

[mrocklin]

This seems like a reasonable idea. It has come up a few times. I think that someone, maybe @jiffyclub maybe implemented this somewhere?

The negative side of this is that it's a hard decision to reverse. That shouldn't really stop us though.

No strong opinions on the name. tlz sounds fine. Is suspect that others will swing by with naming suggestions.

[jiffyclub]

My solution for this is here: https://github.com/jiffyclub/zbox

[eriknw]

zbox is indeed a simple and convenient way to get toolz or cytoolz: it's entire usage is summed up by from zbox import toolz.

tlz is different. It looks like toolz: you can do from tlz.curried import *. If we include it as part of toolz, it becomes that much more convenient to use. We can customize tlz to our tastes, such as always pulling some functions from toolz, and making sure file introspection such as In [2]: tlz.functoolz?? displays toolz.functoolz. @jiffyclub, I welcome your thoughts and if there's any way we can customize tlz (name?) to match your tastes.

[eriknw]

After revisiting this, I think this is a good enhancement and I'd like to merge it. @mrocklin @jcrist, thoughts? Anything other than pipe that you'd like to always come from toolz?

[jcrist]

Sounds good to me. Other pipe-like things might be nice as well (compose, juxt, etc...), but I don't have any real strong opinions about that.

[mrocklin]

No strong thoughts from me.

On Oct 6, 2016 19:03, "Jim Crist" notifications@github.com wrote:

Sounds good to me. Other pipe-like things might be nice as well (compose,
juxt, etc...), but I don't have any real strong opinions about that.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#314 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AASszCJFmXeEBhDKQfH-9mABWEswS1Xbks5qxX45gaJpZM4Iz1th
.

[eriknw]

tlz is going in!