-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up PAM250 for CPTAC #134
Comments
Setting |
Could you profile it with that setting? lru is super fast so I'm surprised it would be the bottleneck.... even things like getting the string from a list could be taking longer. If it is indeed the slow point, there are some adjustments we can make, but they'll be a little more involved. And aren't you already running parallel calculations at another level? |
Currently I'm only running parallel while gridsearching but not during fitting. Good news is that the model seems to work with PAM250. I used a reduced version of the CPTAC data set (~3000 peptides, all patients) and the model strictly improves and ultimately fits. Now it's only a matter of speeding it up. This is the profile: |
This is what I changed but it didn't seem to speed things much more. I also tried calculating the cluster assignments in parallel with |
Are your sequences in a defined and unchanging order at some point? Because you could make an n_seq by n_seq matrix once, then never recalculate the PAM distance again... |
Hmm interesting! The order of the query sequences is the same as in the mass spec data set but the order within each cluster is changing after every iteration... However, I think I should be able to get around this. Otherwise, instead of an n_seq by n_seq matrix, I could collect this information in dictionary form |
I implemented what we discussed above and it sped up the code a lot when using a fraction of the data set, but when I try to calculate the scores matrix with all 42k, it breaks Aretha... As expected, once the pam250 scores matrix is calculated, the model runs much faster. Something strange I've noticed, is that after running |
A few things:
After these changes, let me know how I can run this. I can check one or two other items. |
I think it's ready. I'll commit these edits in #129. To run this, you just have to go to the notebook |
Take a look at my most recent commit on #129. There were a couple (really hard) things to change:
Good news is 30,000 sequences now takes ~2 min. 🤩 FYI, that |
P.S. Because this requires python v3.8, PAM250 will not build on Jenkins for now. That's fine—we can override the build checks. |
Wow, this so cool! Thanks! I'll take a look now. |
No description provided.
The text was updated successfully, but these errors were encountered: