Merge Nathan's code.

[danielballan]

FYI, @nkeim , Tom Caswell is over my shoulder, and we're serious about merging trackpy in the next few weeks. We're going to reconcile my (Dan's) branch first and then yours. I'm also going to pull in all my feature-finding, tests, and analysis code. See related issues.

"Easy"

• Allow access to nonrecursive_link via keyword link_strategy.
• Add KDTree alternative. Access HashTable vs. KDTree via keyword neighbor_strategy.
• Make Tom document this.

Hard

• Add Nathan's linker.
• Allow it to work with numba if numba can be imported; otherwise, without. (Maybe this just works, once we add the function.) Cherry-pick if this is practical, and either way, attribute Nathan in the README.
• Make sure Nathan's API is supported.

[nkeim]

Dan, it was great to meet you yesterday and I'm very excited about this merge.

As I said, I seem to recall that switching to cKDTree was more of a "point of no return" in terms of making a merge difficult. It's good that that switch can make a big difference in performance, at least for some users. The actual numba components are wrapped to make them work more or less like the pure-Python versions.

Please let me know if you get stuck! Thanks - nathan

[danielballan]

Yes, I'm glad we ran into each other, and I am sorry I had to depart early. I am collaborating with the Stebe lab and visiting every week or two -- I'll get in touch before my next trip up there and see if we can chat for awhile.

[nkeim]

Definitely! I'll be right downstairs from you. BTW, I really like the notebooks that are linked to in README.md now.

[danielballan]

Thanks. I have looked over your efforts along the same lines, and I like that we are using pandas in the same way. Are many others in your group using trackpy and your related projects? The documentation looks thorough.

[nkeim]

I'm glad you think so! Yes, trackpy and runtrackpy are used by a couple other people here; Python has pretty much taken over as the default environment. And pandas has changed lives here :). I don't know of anyone else using philatracks heavily except for g(r), but it is a bit more specialized.

I think we agree that it's really important to make the out-of-the-box experience as fun and easy as possible for this software. I wonder if we ultimately want to make trivial datasets available as part of the tutorial. For example, a short video of flow in a microchannel with tracer particles.

[danielballan]

Sounds great. There is a trackpy/tests/water/bulk-water.mov but it's not very pretty. A variety of short, small videos would be neat. Do have one or two you could add? Say, trackpy/examples/video? Ultimately something like import trackpy.examples.sample_features would be cool too.

Be advised that your branch of trackpy predates an important fix of the memory feature from awhile back. See the associated test for an explanation of what was coming out wrong. You did merge an earlier fix, also related to memory, but the second one arrived after you latest commit.

[tacaswell]

I would be wary of loading the repo up with large binary files. It can be pretty awful on slow connections as it is.

It might be worth splitting the demonstration work/template notebook off into a separate repo (trackpy-demo).

[nkeim]

Good catch. AFAIK no one is relying on memory support in my fork, but that's bad. I think that there is no point in backporting the fix; I should put that effort into helping complete the merge and updating runtrackpy.

[nkeim]

Agreed, Tom. I was thinking that the videos could be hosted on someone's group's website. But a separate repo for the movies and the tutorial notebooks could be even better.

[danielballan]

Sounds good. IPython-style tutorials can also be written into sphinx (a la the pandas docs) but I think self-contained tutorial notebooks might be more useful, especially if they come with videos.

ReadTheDocs is up and running, btw.

[nkeim]

The numba-accelerated subnet linker is now merged, at nkeim/trackpy@e4e5106 . The numba subnet code passes the standard tests, and the whole thing also works in an environment without numba installed. Note that as with KDTree, numba is not the default strategy in this version.

Before I make a pull request, I'd still like to try it on some real data, and get some sense of the performance improvement.

[danielballan]

Nathan, is there anything else from your codebase or runtrackpy that belongs in trackpy proper? If not, we'll close this.

[nkeim]

Nothing else to add!

As far as superseding runtrackpy is concerned: is your link_on_disk.py in good working order? (That was the whole genesis of runtrackpy in the first place.)

Thanks!!

[danielballan]

It works on mr. I have to do some minor things to make it work on trackpy, mostly removing unnecessary code. I'll fix it and document it tonight.