Release on PyPI

[pfmoore]

It would be much easier to depend on scandir in my own code if it were registered on PyPI. Even better would be if you published wheels as well. Is there any reason not to release it?

[benhoyt]

The reason is that it's not really battle-tested yet, and the C code is far from complete (even if working at a basic level). I do hope to do some more work on it, but it may not be for a little while...

[poupas]

Ben, could you please elaborate a little on what is missing from the C code?

It might allow others to eventually contribute with some code :)

Thanks

[benhoyt]

Sure -- the main thing is the C version is currently kind of hacked in there and builds a list of results (name_stat tuples on Windows, name_type tuples of Linux) rather than returning a generator which yields results. The slowish Python version, on the other hand, yields results as it goes.

[tjguk]

I do have a fork which started to produce an iterator, but it kind of
fell apart and then got pushed further and further down the to-do list.

For the idea to be any real use, though, I believe it needs to iterate
at C level. (Obviously, it would work without that but...)

[benhoyt]

Thanks, Tim. Actually, it is quite a bit of use without that -- even when building a list, you still get all the speed advantages of scandir returning stat-like info directly from the OS, you just don't get the advantage of it being a generator.

[tjguk]

Of course it is: poor wording on my part; sorry. In any case, have
C-level iteration would certainly be a cherry on the cake.

[pfmoore]

The reason I got interested in this again is that I have a use case for a fast directory scan that returns stat data (a large directory tree comparison program) so maybe I'll be able to contribute some code. It's been a while since I did any Python/C API coding, should be fun :-)

[benhoyt]

Sure -- if it's better than what we've got, I'll definitely include it. Bonus points if it compiles for both Python 2.x and 3.x

[poupas]

Excellent :) It would also be awesome to clarify the whole unicode paths stuff.

I have a minor change in my fork to allow scandir to accept and return str (bytes) objects in Python 2, as it is needed to work around paths which are byte strings with unknown/invalid encoding (e.g. latin-1 paths mixed with utf-8 encoded paths...). This seems to be a Linux-only requirement. Windows works happily with unicode strings and OS X sanitizes the paths so that they are valid utf-8.

[tonyandrewmeyer]

I don't think you need to have it "battle-tested" or the C code complete to make it available on PyPI - just clearly mark it as pre-release, so that people are aware that there may be issues, incompatible changes coming, and so on. It would really be a huge benefit for installation.

If you do really want more done first, would you be able to open issues for each thing you want done first and reference them here? Then I could submit PRs for each of them :)

[benhoyt]

This is now registered on PyPI: https://pypi.python.org/pypi/scandir -- I don't think I'll do anything like wheels till it's stabilized.

[tonyandrewmeyer]

Is there anything that can be done to change your mind about this? (Like any sort of patch?).

You're already bumping version at various points - if you could just create a package whenever that happens and upload it to PyPI, that would mean that pip install would work (no need for wheels). Or if you'd prefer to host releases on GitHub, you can use the "urls" tab to tell PyPI where downloads are found (choose "do not extract" at the top, then add the URL manually, ensuring that the hash is present). That'll require "allow external" to be set, but would still make the package pip installable.

If the goal is inclusion in 3.5, then having the package available from PyPI will help, because that's where most people install things from, so you will better be able to show wide usage.

[benhoyt]

Yeah, that's a fair call. No, I'm not opposed to this -- I just don't want to learn wheels at this point. :-) So for pip install to work, I just need to do a "setup.py sdist" or similar and upload the zip to PyPI?

[pfmoore]

Correct.

Just FYI, learning wheels isn't hard, just install the wheel project and do setup.py bdist_wheel. Then upload the .whl file. The only issue is that like any binary format, for a C extension you need to provide separate builds for each Python version and architecture. (win32/win64, wheels are mainly useful on Windows at this point).

[benhoyt]

Thanks. Yep, happy to upload a source zip to PyPI (I'll do that in the next couple of days). However, I'm going to leave the wheels / binary builds at this point.

[benhoyt]

Uploaded v0.4 files (.zip, .tar.gz, and Windows 64-bit installer just for fun) to PyPI.