Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log file name unpredictable on keyboard interrupt #377

Closed
campenr opened this issue Sep 26, 2017 · 5 comments

Comments

@campenr
Copy link
Contributor

commented Sep 26, 2017

Hello

Because mothur creates its own name for the log file, and then only renames that log file to what the user specified with set.logfile() on completion of execution, a Keyboard Interrupt results in the log file bearing the random mothur generated name, not the user specified name. This causes any downstream processes relying upon a predictable log file name to break.

It seems like it would be most easily solved by setting the name of the log file to the user specified name from the start. This would then also allow things like tail -f <mothur.logfile> to read the log file in real time. It would also (depending on the implementation of writing to the log file) allow for setting the log file to /dev/null to suppress the creation of a log file at all, which would be nice to have in the absence of an option within mothur to do this directly.

Cheers
Richard

@mothur-westcott mothur-westcott added this to the Version 1.41.0 milestone Sep 26, 2017
@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2017

Thanks for the suggestion!

  • We are adding a compile time option to set a static log filename.

ifeq ($(strip $(LOGFILE_NAME)),yes) LOGFILE_NAME="\"mothur.logfile\"" endif

  • You can set the static log file name to "dev/null" to suppress the writing of the log file at all.

ifeq ($(strip $(LOGFILE_NAME)),yes) LOGFILE_NAME="\"dev/null\"" endif

  • You can set a run specific log file name by running the set.logfile command as the first command in a batch or script.

sarahwestcott$ ./mothur "#set.logfile(name=./sarah.run_1.logfile);summary.seqs(fasta=final.fasta);"

Note: You can compile mothur with the default log file of "dev/null", and then set a run specific name for any runs you wish to have a log for.

@campenr

This comment has been minimized.

Copy link
Contributor Author

commented Dec 1, 2017

Just to clarify, with this update will the set.logfile command now name the log file to what the user specifies from the time of creation of the log file instead of just renaming the log file after it has finished being written to as is the current behavior?

@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Dec 1, 2017

Mothur buffers the log, until the first command is processed. If the first command is set.logfile, then mothur creates a log file with the user specified name. (no renaming) If the first command is not set.logfile, then mothur will create a log file with the name specified at compile time(this could be dev/null or mothur.logfile). If no name is specified at compile time then a random name is generated.

@campenr

This comment has been minimized.

Copy link
Contributor Author

commented Feb 13, 2018

I've been testing this as is implemented in the current Threads_373 branch as of commit b111b6e, and am finding some interesting results.

  1. Mothur seems to not play well with writing to /dev/null:

When built with the default Logfilename ?= no flag, set.logfile(name=/dev/null) fails with the message /dev/ directory does not exist or is not writable.. This seems to be a permission error as mothur can write to /dev/null IF run as sudo. This work around would not be doable for all users if they don't have administrator permissions. This behavior is odd as all users have write permissions to /dev/null so I am wondering if it something to do with the way in which writing to the logfile is implemented in mothur...?

Side note specific to this error, when the logfile name is invalid, (such as when using /dev/null as a non-sudo user) mothur silently (in that it does not raise either a warning or an error) assigns the default logfile name, which in this case is a random number. I think the more desirable behavior would be to raise an error, or at least a warning, as if the user is specifying a logfile to write to, and mothur is unable to comply, the user would probably want to know and to rectify the problem, rather than mothur use a different logfile name. Otherwise the command set.logfile appears to work as desired in this build.

  1. Configuring default LOGFILE_NAME during build does not seem to work:

When using mothur built with Logfilename ?= yes flag and setting the logfile name default as LOGFILE_NAME="\"<anything here>\"", mothur does not seem to use that, instead still generating its own random logfile name. However, in this build, when I try to use set.logfile(name=my.logfile) I get an error [ERROR]: Could not open my.logfile which makes sense in that this file does not exist because mothur never made it... I have tried setting several things as the LOGFILE_NAME when building mothur including /dev/null and test.logfile all with the same result in that mothur still uses its own numbering system for logfiles and set.logfile is broken.

mothur-westcott added a commit that referenced this issue Mar 6, 2018
@mothur-westcott

This comment has been minimized.

Copy link
Contributor

commented Mar 6, 2018

Thanks for the feedback. Here's the rework:

We are adding a compile time option to set a static log filename.
ifeq ($(strip $(LOGFILE_NAME)),yes) LOGFILE_NAME=""mothur.logfile"" endif

You can set the static log file name to "silent" to suppress the writing of the log file.
ifeq ($(strip $(LOGFILE_NAME)),yes) LOGFILE_NAME=""silent"" endif

You can set a run specific log file name by running the set.logfile command as the first command in a batch or script.
sarahwestcott$ ./mothur "#set.logfile(name=./sarah.run_1.logfile);summary.seqs(fasta=final.fasta);"

With this new setup, I would recommend compiling with silent and setting a standard name for the logs you want to keep. For example you could capture the log for a specific command within mothur as follows.

mothur > set.logfile(name=silent) - surpress logging
mothur > align.seqs(fasta=amazon.fasta, reference=silva.v4.fasta) - run align.seqs command without logging
mothur > set.logfile(name=summary_Align.seqs) - add log to grab results of summary file
mothur > summary.seqs(fasta=current)
mothur > set.logfile(name=silent) - silence logging
mothur > summary.seqs(fasta=final.fasta)
mothur > quit()

Would result in a summary_Align.seqs logfile that contains:

mothur > set.logfile(name=summary_Align.seqs)

mothur > summary.seqs(fasta=current)
Using /Users/sarahwestcott/desktop/release/amazon.align as input file for the fasta parameter.

Using 4 processors.

	Start	End	NBases	Ambigs	Polymer	NumSeqs

Minimum: 1964 13425 259 0 3 1
2.5%-tile: 1964 13425 261 0 3 3
25%-tile: 1964 13425 277 0 4 25
Median: 1964 13425 277 1 4 50
75%-tile: 1964 13425 277 2 5 74
97.5%-tile: 2393 13425 278 7 8 96
Maximum: 3049 13425 279 13 8 98
Mean: 1988 13425 276 1 4

of Seqs: 98

It took 0 secs to summarize 98 sequences.

Output File Names:
/Users/sarahwestcott/desktop/release/amazon.summary

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
2 participants
You can’t perform that action at this time.