error with qiime micom build and qiime micom grow commands #1
Comments
|
Oh no, I'm sorry it didn't work as expected I will investigate and get back to you. I might need some time to check what is going on with Gurobi since I don't have access to it at the moment but maybe it related to something else. |
|
Hey I tried all the samples you mentioned with cplex and Qiime 2 2019.10 and it all went fine. However, as mentioned in the forum it was a bit of work installing the package here since I had to upgrade a lot of dependencies by hand and needed to install pyarrow with pip due to a conda conflict. So I would recommend using the dev environment until the new Qiime 2 version is released (scheduled for this month). I also encountered errors with the visualizations and this is now fixed in version 0.5.1. The crossover errors are probably due to some problem with Gurobi. I'll try to get my hand on it and investigate more. For now the my recommendation would be to use CPLEX. I will update the tutorial to mention those pitfalls. Planned changes
|
|
Dear Christian, thanks for your support. I updated to version 0.5.1 and installed CPLEX. However, I still get the same errors. Since I now have both Gurobi and CPLEX installed as my QP solver - is there an easy way to force micom to use CPLEX over Gurobi? I also remember that there were several issues when I installed micom the first time. Maybe I did not solve all issues back then correctly? For now I guess I'll wait for the next qiime2 release (2020.2) and then give micom a new try... |
|
Hmm not in the current version of q2-micom. I will update micom to prefer cplex. If you installed q2-micom in its own environment you could simply uninstall Gurobi there. Your local Gurobi license will not be touched by this and remain intact. If you don't want to wait you can also create a developer environment for Qiime 2 which is pretty much the new version at this point. You can find instructions for this here: https://dev.qiime2.org/latest/quickstart/. |
|
I´m having the same problem using Gurobi and CPLEX. Using the tutorial data I got warnings in all samples, it took 2 hours using 16 cores:
After that I got the
|
|
Yes, I saw the same thing. There was a bug in micom where it would not use Gurobi when installed. However that would only affect environments where cplex is not available. This is fixed now in micom so you could try just to update micom. What is the output of import cplex
print(cplex.__version__)when run in Python of your Qiime 2 environment? |
|
The cplex version: |
|
Hi Christian, Successfully built q2-micom Thanks for your help! |
|
Yes, that is something I need to circumvent in the install. The problem is that bioconda and conda-forge both have tqdm and bioconda has a very outdated one. Bioconda has the priority in the current installation instructions which is wrong. You can try to fix it with |
|
Okay, I updated the installation instructions in the tutorial and Github Readme. Could you try with those in a fresh Qiime environment? It worked for me on my Mac, but would be good to see if it does for you. |
cdiener
added
the
ready
|
I updated I still have the error:
and
I am working in Centos 7.4 |
|
Did you run |
|
It works for me 🎉 |
|
This seems to be fixed. Feel free to reopen if the problems persist with the new package versions and installation instructions. |
|
Hi, Using the tutorial dataset (crc_table.qza, crc_taxa.qza) with the micom build command: (1/2) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2 Using the moving pictures tutorial dataset with the micom grow command: Plugin error from micom: All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in and finally with my own dataset, again with micom build: Plugin error from micom: Length mismatch: Expected axis has 15 elements, new values have 8 elements Sorry for bothering you again... |
|
Hi, similar as in the comment before you will need to rerun How did you download the data? Could you try to download it again making sure to use the "Download" button in the upper right corner? I have already seen those error pop up in broken Qiime 2 environments since an error in the plugin system often yields those as well. Does this also occur in a fresh Qiime 2 environment? Does running The error on your own data does look like a bug. Unfortunately, it's hard to me to say what is going on without seeing the artifacts. How were those tables generated and what classifier did you use to assign taxonomy? Could you paste the first few lines of your taxonomy data? |
|
Hi, Regarding my own data: They were created with a qiime2 versions from 2018 and 2019 using DADA2 for denoising. If you think there might be a compatibility issue with older qiime2 artifacts I could dig deeper into it. Taxonomy was assigned with a classifier based on Silva132 with OTUs clustered at 99% and trimmed to V4 515f-806r primer regions. Here are the first lines of my taxonomy file: Feature ID Taxon Confidence thanks for your help! |
|
That actually looks good. I also used SILVA for the tutorial data, so that should work. Did you really cluster the OTUs after DADA2? This is usually not necessary since DADA2 is in itself kind of a (better) clustering algorithm for 16S data. But this should not be issue here. What I can see from your taxa is that it is likely not a human gut sample and there are many more taxa ranks than I would expect. When I use the SILVA classifier I never get more than 7 taxa ranks (kingdom, phylum, class, order, family, genus, species). So this is surprising to me: That is likely where the error comes from since that taxon has 15 ranks and q2-micom assumes at most 8. I will investigate a bit how to fix that cleanly but can probably add a dirty fix that just uses up to 8 ranks and gives a warning. I'll get on that... I should note that most of your taxa are probably not contained in AGORA which is pretty specific to the human gut. I am in the process of providing the CarveME model DB (>5K genera) for q2-micom as well but this will still take a little bit. |
cdiener
removed
the
ready
|
Hi, Next, I will use a different test dataset (due to the gut specific model database) and make sure I'll only have 7 levels in my taxonomy artifact. Unfortunately, I had another error after successfully building the model with micom grow and the crc tutorial dataset: Plugin error from micom: All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in Looking forward to get my hands on the CarveME model DB as soon as it is available and the above errors could be fixed... |
Okay this makes more sense.
This is odd. If you run this with the python -c "import gurobipy; gurobipy.Model()"
gurobi_cl --versionand your cplex installation with: python -c "import cplex; print(cplex.__version__)" |
|
I asked on the Qiime 2 Forum and they recommend to use a revised SILVA classifier that will only give 6-7 level taxonomies -> https://forum.qiime2.org/t/silva-138-for-qiime2/12957. This would probably fix your issue with your own data set. |
|
Hi, sorry for my late response (a bit difficult to find time for testing due to corona virus restrictions...). Thanks for flagging the post on silva-138! I re-classified my sequences and now I can make models with all the tested datasets (no more length mismatch errors) :-) However with micom grow I still get the following error (irrespective if it is the tutorial data or my own data). I also tested my cplex installation with python -c "import cplex; print(cplex.version)" and then I get this: I assume there is a problem with cplex and micom here (maybe specifically in my conda environment), despite I followed each step in your installation tutorial and did not get any errors... If you have any ideas, I'm happy to keep on testing... |
|
This is pretty odd. I have tried it with that cplex version and the the tutorial data and it works in my environment. It could be a problem with the environment setup or with the model building. |
|
Hi, |
|
Bit at a loss here since I also use Debian and have similar versions. Would you mind providing the commands/scripts you are running? Or did you paste the tutorial commands verbatim? |
|
Yes, more or less. qiime micom build --i-abundance table-dada2.qza --i-taxonomy taxonomy.qza --i-models /home/cluster/o_mahnerta/databases/micom/agora_genus_103.qza --p-cutoff 0.0001 --p-threads 12 --o-community-models models.qza --verbose qiime micom grow --i-models models.qza --i-medium /home/cluster/o_mahnerta/databases/micom/western_diet_gut.qza --p-tradeoff 0.3 --p-threads 48 --o-results growth.qza --verbose Plugin error from micom: All numerical optimizations failed. This indicates a problem with the solver or numerical instabilities. Check that you have CPLEX or Gurobi installed. You may also increase the abundance cutoff in |
|
That looks good. Do you think you could share your models.qza file for the tutorial data with me? That way I could check if something looks amiss. |
|
Sorry the file is too big to put it here.
|
|
So I ran qiime micom grow --i-models ~/Downloads/crc_models.qza \
--i-medium western_diet_gut.qza \
--p-tradeoff 0.3 \
--p-threads 4 \
--o-results crc_growth.qza \
--verboseAs a last resort cold you also try running the command with If that does not work this will unfortunately be hard to investigate but a good initial check would be to run some of the MICOM test data in Python to see if there are some problems. Also make sure you did install a full license for cplex (can be academic) and not the community edition provided on conda (limits the number of variables). For instance you could run the following in your Qiime python: import micom
tax = micom.data.test_taxonomy()
com = micom.Community(tax)
sol = com.cooperative_tradeoff(fraction=1.0)
print(sol.members) |
|
Hi, after a long time I got back to test micom again. And you were right - my issue was related to the community edition of cplex. After installing the academic full license qiime micom grow worked out finally. Thanks for all your help! |
|
Awesome. Glad that worked. I should make a note in the installation instructions that the community edition won't work. |
Dear Sean and Christian,
I'm very curious about micom and I tested it as a new plugin for qiime2. However, I experienced several issues:
First I tried to apply the tool on my own example dataset, but received the following error with the qiime micom build command:
Plugin error from micom: Length mismatch: Expected axis has 15 elements, new values have 8 elements
Ok then I thought I should stick to your example data covered in your tutorial (https://github.com/micom-dev/q2-micom/blob/master/docs/README.md)
However, for the crc_table.qza and crc_taxa.qza files I received the following error with qiime micom build:
(1/2) Invalid value for "--i-abundance": 'crc_table.qza' is not a QIIME 2
Artifact (.qza)
(2/2) Invalid value for "--i-taxonomy": 'crc_taxa.qza' is not a QIIME 2
Artifact (.qza)
Hm, then I thought let's try the small dataset of the moving picture tutorial (https://docs.qiime2.org/2019.10/tutorials/moving-pictures/) and the qiime micom build command worked! :slight_smile:
Nevertheless, with the qiime micom grow command I got another error:
Plugin error from micom: 'NoneType' object is not subscriptable
I guess this was due to several warnings I received during the execution that qiime micom grow could not solve cooperative tradeoffs for many samples.
So I was wondering how to proceed from here. Would be great if you could comment on those errors or maybe check the artifacts of the crc files, or recommend other suitable test datasets to get an idea how to get micom work...
By the way I used qiime2 2019.10 and the gurobi license.
Thanks for any support!
The text was updated successfully, but these errors were encountered: