Add include_path option for dask.bag.read_text

[gyf304]

• Tests added / passed
• Passes black dask / flake8 dask

[jakirkham]

@mrocklin, would you have time to review this or recommend someone who could?

[gyf304]

I'm internally debating if include_path should return a tuple of (bag, paths) like dask.bytes does, or return a bag of (line, path) like what dask.dataframe does. Not sure which way is more useful.

[TomAugspurger]

@gyf304 just FYI, when you force push to the branch we don't get notifications and it makes things a bit harder to review. Can you just push additional commits? We squash on merge.

internally debating if include_path should return a tuple of (bag, paths) like dask.bytes does, or return a bag of (line, path) like what dask.dataframe does.

Can you include documentation / examples for what this does? That may aid in determining what is more natural. My initial reaction is that a keyword probably shouldn't change the return type of a method (from Bag to Tuple), though it's hard to say.

[gyf304]

Sorry about the force push, was used to the workflow of gerrit of doing git commit --amend. Will directly commit next time.

Example for first approach

>>> bag, paths = dask.bag.read_text("./test/*.txt", include_path=True)
>>> paths
["/home/.../test/1.txt", "/home/.../test/2.txt"]
>>> len(paths) == bag.npartitions
True

Example for second approach

>>> bag = dask.bag.read_text("./test/*.txt", include_path=True)
>>> bag.take(1)
(("hello", "/home/.../test/1.txt"),)

[TomAugspurger]

Do you have a suggestion for which is more useful?

I would think that the first is relatively easily accomplished by

paths = glob.glob("*/test/*.txt")
bag = db.read_text(paths)

The second is harder to achieve without the include_path keyword.

[gyf304]

I agree. This also mirrors well with the include_path_column option in dask.dataframe.read_csv
https://docs.dask.org/en/latest/dataframe-api.html#dask.dataframe.read_csv
I already updated the patch to the second behavior after the same thought process.

[TomAugspurger]

I think it'd be a bit easier to understand with something like

>>> b.pluck(1)
(('first line of the first file\n', 'myfiles.0.txt'),)

Why is the ... present in your output? Are the paths relative or absolute?

[gyf304]

Think it should be

>>> b.take(1)
(('first line of the first file\n', '.../myfiles.0.txt'),)

Paths are absolute. The ... is just ellipsis for path. Open to suggestions for alternatives to ...

[gyf304]

Maybe something generic like /home/dask/myfiles.0.txt?

[TomAugspurger]

Thanks @gyf304!

[SultanOrazbayev]

This is great, thanks @gyf304!