New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline error #55
Comments
Issues that I see with your command: Could you please attempt to run the following minimal test command:
edit: easier to read |
Hi, I ran the minimal test command but still got errors:
Do I need to install tools used in the nf pipeline under my home directory where nextflow was installed? According to my understanding, this's seems to be unnecessary. |
Seems to me like a docker-related issue. For running nextflow with containers you have to have singularity or docker installed, you specified docker to execute the container. Is docker installed? |
Yes, this is a Docker installation / configuration issue. Please follow the Docker documentation first on how to start/test your local Docker installation for example here: https://docs.docker.com/install/linux/docker-ce/ubuntu/ |
Hi, thanks! The docker was installed and tested as suggested. Here's what I got when I typed in the test command:
But if I typed in the second test command, there's a permission deny problem. Could this be related to the error?
|
Follow these steps as well to solve the permission problem: |
Does it work for you now? |
Hi, sorry for the late reply. After proper configurations of Docker, I ran the test commands without any problems. Next, I ran the command block you suggested.
The pipeline proceeds to qiime_demux_visualize with no further actions. I could find the html-based interactive sequence quality plot under the results folder and determine the truncate length for my reads. But I was not able to find the imported qza file to be supplied to the arguement Many thanks! |
Hi, you stumbled there about an inconvenience, I have opened an issue with an easy fix for the next release.
Hope that helps |
Hi, I started over again but I had a problem in making the SILVA classifier.
Here's the error message:
I was asked to check the log file but I couldn't make sense of it. |
Hi, this seems to me like a user right problem, indicated by this line in the log: @apeltzer: any idea how to tackle this issue? |
Sounds a bit weird to me too - can you use the Singularity container please and try again?
If that doesn't work, we can at least easily check why the permissions don't fit. |
Thanks! |
Hi, I may have figured out the problem. The silva classifier requires a much higher memory usage than the greengenes does, which in my case needed 35 GB of memory that exceeded the available system memory (31 GB). Other qiime2 users have also reported similar issues. The following is the error message when I didn't restrict the usage of memory and cpus.
I resubmitted the workflow by skipping the taxonomic classification |
Skipping taxonomic classification should prevent this to happen, because this process shouldn't be executed at all with Since you seem to use the 515f/806r primer pair you can download the qiime2 classifier here: I am a little surprised that nextflow mentions the 35Gb specified in conf/base.config, I'll look into that as well, since in some cases this process takes actually significantly less memory. |
I tested the pipeline on my laptop (4 cpus, 16Gb RAM) with the following parameters: I resumed the pipeline with the parameters I cant find here any bug or unintended difficulty. Training the SILVA 132 classifier requires ~33 Gb memory, no way around that. Alternatives are to use a pre-trained classifier (has to be trained with same primers as in PCR) as I explained above or train a smaller one (using the non-documented and not recommended hidden parameter edit: with 30 Gb memory you will be limited by sample and read numbers in your analysis. The maximum memory that I encountered yet were 63Gb for classification of ~100 highly divers samples. Just to clarify that. |
Hi, I used the pre-trained SILVA132 classifier from the QIIME2 website and proceeded with the taxonomic classification:
However, the taxonomic classification task was killed as it exceeded the maximum run time.
A similar issue was also reported on the QIIME2 forum regarding the memory usage problme with the silva classifier. To find out if the system memory is enough for the task, I trained the silva classifier and used it for the taxonomic classification within QIIME2-2018.11 using the same workstation where I ran the nextflow. All the command lines below worked without problems. The training of classifier took 1h 41 minutes and the taxonomy classification took 17 minutes. The rep-seqs-dada2.qza contains 1399 unique representative sequences. As you've pointed out, it shouldn't require massive memory usage and take more than 2 hours to be finished. I can't figure out why it's a problem when I use the nextflow pipeline.
|
Could you try the former nextflow command with singularity instead of docker? I hope that gives better results. Training the classifier should always need ~33 Gb RAM, maybe you succeeded with your latter commands because its an edge case and swapping helped. |
Hi, I tried the former command again using the docker image and it worked all the way until beta-diversity, which displyed a similar error message:
I also ran the former command block using the singularity image but I got a different error:
|
The docker container indeed seems to have problems with your machine. @apeltzer are we going to attempt to fix this or should we maybe highlight somewhere that only singularity is supported? With singularity simply the metadata file is not found. Could you verify the path? I have already implemented an early check for file existence for a future release. |
Hi, the metadata file path was correct. Anyway, I changed the file path and ran the workflow using the singularity image.
Here's another error:
|
I see two possible problems with your command: change the line (see the last "/") and (see the space before last "\") Please make sure your spelling is correct. |
Hi, the error in computing the beta-diversity using the docker image was due to the wrong metadata format, which was made for qiime2-2018.11. After reformatting of the metadata, the pipeline was finished without any problems. I also followed the instruction in the docker documentation to fix the warning message: I tried to repeat the workflow using the singularity image but without sucess. I'll open a new issue. ps: |
Good to hear that it worked out so far. Feel free to open a new issue when needed. |
hi @yanxianl! Could you please tell me what was your error in the metadata file? I am getting the same error with the beta-diversity calculation. |
Hi, my metadata was made for QIIME2-2018.11, which is not compatible with the QIIME2-2018.6 implemented by the current nextflow pipeline. I reformatted the metadata according to the QIIME2-2018.6 documentation and it worked. |
Hi,
I tried to run the nextflow pipeline for a small dataset but encountered an error.
The commands I used:
The error message:
My java version:
Regards,
Yanxian
The text was updated successfully, but these errors were encountered: