Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bus Error: 10 on cluster.split run on Mac #644

Open
EleftheriaCha opened this issue Jul 30, 2019 · 19 comments

Comments

@EleftheriaCha
Copy link

commented Jul 30, 2019

Hey there,

Im fairly a new mothur user, I have been trying to analyse 16S data with mothur and it seems I always fall on the same error. I did tried to run with various mothur versions (1.42.1/1.42.3/1.41.3) and I always get he same error. Im using mothur for Mac and the error is BUS ERROR 10 on cluster.split command.
I have tried this with test data and actual data but still get the same error over and over again.
I would be grateful if u could help me… I
do have the log files from all the runs I did and the batch file I’ve used. Please tell me which is more helpful for you and I will send it through.

Thank you in advance.
Best,
Eleftheria.

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

It could be a memory issue. Are you getting this error with the MISep_SOP dataset, https://www.mothur.org/wiki/MiSeq_SOP? How big is your dataset? How much memory do you have on your MAC? Could you post your log file?

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Jul 30, 2019

Hey Sarah,

Thank you for the quick reply.

It shouldnt be a memory issue as I am using an external 4TB hard drive and I also tested the command with a decreased number of processors (I have max 12 and I run as low as 4)

The original dataset is quite big 94 samples however I have tested using only 2 samples from the original dataset and the error remains the same.

I am also currently testing the same commands and same file on a windows just to see if its a MAC only error I get or not.

Yes I can attach the log files for the original dataset and the test. Also I saved the terminal output as a text file for an overview. I am more than happy to share these but unfortunately I cannot upload them in their current file format (.txt) any advice how to upload or any alternative way to share?

Thank you in advance.
Eleftheria.

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

The memory I am referring to is not the hard drive, but the RAM. "Both RAM and hard drive memory are referred to as memory, which often causes confusion. RAM stands for Random Access Memory. ... When your computer is turned on, it loads data into RAM. Programs that are currently running, and open files, are stored in RAM; anything you are using is running in RAM somewhere." You may be able to write the distance matrix to the hard drive, but not able to store it in RAM. For example, I have a 2TB hard drive, and 32Gb of RAM. I can create a distance matrix that is greater than 32Gb, but I can't store that matrix in RAM for processing. Could you compress the the log files and upload them?

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Jul 30, 2019

Another thing to consider when using an external hard drive is the transfer speed. The time to transfer data to and from an external hard drive can increase the processing time of mothur's commands.

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Jul 31, 2019

Dear Sarah,

I have checked for RAM as well, my pc is 64gb RAM and I monitored the memory at the very begining when i started with mothur, and i had no problem with it. The only difference is that then I used the computers internal storage memory whereas now i switched to the external.
i have thought that perhaps writing on the external may affect the processing. I might repeat a run using the pc's internal memory and see how it goes...

Also the test I did on windows it passed succesfully!

I created a mix zip file with the log files from mothur where i had the error on MAC, the successful log from Windows and also the terminal output from the MAC runs.
LogFilesMix.zip

Let me know if you have any further ideas for this issue. I will keep you posted on the additional test I will do running and writing on PC internal and not on the external.

Have a nice afternoon in the meantime :)

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Jul 31, 2019

The log from version 1.42.3 on the mac makes it successfully through the cluster commands. If you send your shared file I can take a closer look as to why the get.communitytype command is failing.

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 1, 2019

From the attached log files I had only the 1.41.3 version that went through to the get.communitytype. I zipped the shared file below.
TestBatch.trim.contigs.good.unique.good.filter.unique.precluster.pick.pick.opti_mcc.shared.zip

My understanding it is that something is not right between the cluster.split to the make.shared file and it is a concequence that the get.community fails but perhaps I am wrong..
Really looking forward to see what you think!

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Aug 1, 2019

It doesn't make a lot of sense to run the get.communitytype command with 2 samples. The default minpartitions is 3 and maxpartitions is 10. The get.communitytype command starts by assigning each partition a random sample. You only have 2 samples. The error is occurring because mothur doesn't have enough samples to assign to the partitions. This command is designed to be run with dataset with samples > 50 samples. I will add a check to mothur to reduce the number of partitions so that this error doesn't happen. In the meantime if you want to run it with 2 samples, set maxpartitions=2.

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 2, 2019

Hey Sarah,

I have tried what you suggested for the maxpartitions=2 and it worked succesfully twice. So then I decided to increase the no of samples thinking that it should pass but unfortunately i got again Bus Error 10 in different commands.

I have zipped the passed and failed log files and terminal commands as for the fail ones did not produce good log file. Keep in mind that i run for 47 samples and 15 samples and updated the max partition accordingly, but it fails long before...
BusError10 20190801.zip

I hope you can understand better than i do why this happens as I do not change the batch file from passing run to failing run, i only update the directory.. :(

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Aug 5, 2019

...
Group count:
1 440923
2 139384
STD 297116
UID 2275990

Make.contigs is only finding 4 groups. I suspect this is caused by how the make.file command is creating group names. Can you check the testbatch.files file to make sure the fastq file pairs are each assigned a unique group name?

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 6, 2019

Hey Sarah, you are absolutely right, i checked all the files i produced now with make.file and i think as i used an _ in my Sample ID it was creating the file wrong all along..
I renamed all the files and I corrected it, i did another test and past but I had a crash again on the make.contigs command for a specific sample.
I am trying to get the log files at the moment so I switched from my personal pc to a work server pc working on Linux version.
I will send you the file with the error as soon as I can get it and zip it. Not sure if it helps you to understand in the meantime but the error sais something like sequence does not exists in both forward and reverse files..

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 6, 2019

Here is a screenshot of what I actually see on the terminal..
image

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 6, 2019

I tried compressing the log files but they are still too big... any ideas how to send them to you? :/

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Aug 7, 2019

Can you run that sample on its own?

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 8, 2019

hey Sarah, apologies for delayed response, its 3 samples that caused me that problem. I isolated them and run all 3 together and it passed perfectly fine. Have a look at the log file also here:
mothur.1565260948.logfile.zip

I did run on Mac version though as I am still running all the rest of the samples on the Linux. The batch i used is the same just updated the directory..
Any ideas why I had this issue on the Linux?

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 9, 2019

So, these 3 samples fail on make.contigs only when I ran the linux version with the following message
image

Any ideas why I get this only on Linux version?

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Aug 9, 2019

Can you send the 3 failing samples to mothur.bugs@gmail.com?

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 9, 2019

Yes Of course will do this in a bit and i can also add the files that do pass to help with troubleshooting!

@EleftheriaCha

This comment has been minimized.

Copy link
Author

commented Aug 9, 2019

Just to let you know I am sending the email to mothur.bugs@gmail.com via a FileSender as the attachments are way too large and it will take some time for the email to be send..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.