Recursion Error (Different from Previous Post) #23
Comments
|
Hi, thanks for the report. You are correct that this is likely a version issue, well spotted. I should fix that up (my fault for only looking at the latest docs and going with RecursionError). Of course the other issue is that you are getting such a thing at all, but then with 1.7 million data points that might just be possible if they don't split too nicely. Still, expected is something like a depth of 21, so something is going seriously astray. If you can share the data I would be interested in trying to figure out what is going wrong exactly. |
|
Now that I've got it running in 3.5, I got a much more detailed error: `Traceback (most recent call last): During handling of the above exception, another exception occurred: Traceback (most recent call last): File "../../../../anaconda2/envs/py35/lib/python3.5/site-packages/umap/umap_.py", line 248 This error may have been caused by the following argument(s):
I don't think I'm allowed to share the data, so I'll keep troubleshooting and let you know if I figure anything out. I'd appreciate if you have any insight from this more detailed error. |
|
That's fine, data is often not shareable. This new error is a pre-error that is again my fault. There is no array checking in place as scikit-learn does yet, which means you can't pass in a pandas dataframe (at least not yet).. To get past this you shoudl be able to do but that may land you back in recursion error land. You can probably just change that by hand yourself in the code if you need to -- I won't be able to get a chance to test a commit a fix for a little while (a PR would be very welcome if you do get the fix sorted). I'll also try and add in scikit-learn's checking to make this work better. |
|
After looking at options I went with the simplest possible thing I could do. Unfortunately I have had issues replicating the RecursionError so I admit my testing that this will actually fix the issue is quite inadequate. I would greatly appreciate it if, when you have available time, you could pull from master, rebuild and reinstall and see if this fixes the problem for you. Thanks again. |
|
The recursion error went away as soon as I changed the input from a data frame to an array (I didn't pull from master to get this result). The job does finish now, granted it takes some time, but the first result I got is really strange. This is with metric="correlation" and n_neighbors=25. |
|
The correlation metric actually has some issues (which I only very recently debugged). Try cosine instead and see if that helps. It certainly is an odd result, but may look stranger due to overplotting. Consider setting the |
|
I might know something about that weird result. I haven't checked in with the latest version of the code for a while so I might be out of date here, but here goes. Briefly: Those results pop up when the learning rate is 'bad' (there could be other causes, this is the cause I know about). Playing with the inputs a and b could help that. In more depth: the program determines a learning rate (or embedding adjustment rate) based on a few values. The user has control over those values to various degrees (layers of abstraction). Using the defaults, the program determines this off of the spread and min_dist values, defaults being 1.0 and 0.1 respectively. From there it passes those values to a function which solves for a and b using a curve fit on an exponential decay. Using the default spread and min_dist, the fitted a and b values are about 1.577 and 0.895 respectively. After that those values--combined with the alpha and gamma parameter (defaults being 1.0 and 1.0) and the embedding data--are used in optimize_layout to determine grad_coeff and grad_d which impacts the embedding adjustments. If you need to use correlation, playing around with those values might help resolve a signal from the noise. |
|
Above is the result for using method "cosine." Looks similar to "correlation." (Groups are NOT labeled - color is random). I ran t-SNE with a subset of 100k data points, and this is what it looks like (groups ARE labelled; color is not random).
Perhaps UMAP would look similar if I only chose a subset of 100k data points? This update is for your benefit. I've moved on from this point. |
|
Thanks for the update, it does look like something is going astray. It may be related to issue #32 . It may also relate to hubness issues. I'm really not sure, but it is something I will continue to look into. I greatly appreciate the feedback, and am sorry that UMAP didn't seem to work for you. Hopefully I'll have some fixes in the future, so please don't write off the algorithm entirely just yet. |
|
I wanted to provide an update on this -- I know you have moved on to other issues, but just in case you run into another project where UMAP might be useful... I believe this issue has finally been resolved. It proved remarkably tricky to get to the bottom of, but was, in the end, a code bug in the SGD optimization of the layout. The latest master branch (v0.2.0+) has a new SGD optimization layout algorithm that does not encounter this issue and should produce much better looking embeddings for large datasets. |
|
Looks resolved. Closing. |
EDIT: Is this just a version issue? RecursionError was RuntimeError before python 3.5.
I'm working with a large data set (~1,700,000 x ~400), and I'm getting the following error:
Traceback (most recent call last): File "umapping.py", line 25, in <module> u = umap.UMAP(metric="correlation").fit_transform(data) File "/users/nicolerg/anaconda2/lib/python2.7/site-packages/umap/umap_.py", line 1573, in fit_transform self.fit(X) File "/users/nicolerg/anaconda2/lib/python2.7/site-packages/umap/umap_.py", line 1534, in fit self.verbose File "/users/nicolerg/anaconda2/lib/python2.7/site-packages/umap/umap_.py", line 559, in rptree_leaf_array except RecursionError: NameError: global name 'RecursionError' is not definedThis is the relevant bit of code:
data = ps.read_csv(args.input, compression="gzip", header=1, sep=',') nrows = len(data) colors = np.random.rand(nrows, 3) # RGB colors u = umap.UMAP(metric="correlation", n_neighbors=25).fit_transform(data)I increased n_neighbors from the default of 15 to 25 to see if that would help, but I got the same error. I do not expect that I have equivalent rows in my data. I am trying to cluster the ~1,700,000 instances in ~400 dimensions. Any suggestions?
The text was updated successfully, but these errors were encountered: