Question on Bismark script for Mox

[yaaminiv]

I wrote a script to run bismark in Mox: https://github.com/fish546-2018/yaamini-virginica/blob/master/scripts/2018-10-12-Bismark-Revised-Parameters.sh

I have a few questions/things I need clarified:

1. I assume I would have to create my SBATCH directory before I run the script
2. How do I upload the C. virginica genome and trimmed files to Mox? The file transfer instructions has information on moving files from Mox. Would I just switch the path to my remote directory and path to directory on Mox?
3. I prepared a bisulfite genome on hummingbird (see this Jupyter notebook). Should I move the prepared genome to Mox, or would it make sense to run the genome preparation step on Mox as well?

[sr320]

1. I assume I would have to create my SBATCH directory before I run the script

You will have to create output/working directory

2. How do I upload the C. virginica genome and trimmed files to Mox? The https://github.com/RobertsLab/hyak_mox/wiki/File-Transfers has information on moving files from Mox. Would I just switch the path to my remote directory and path to directory on Mox?

The wiki needs updating but actually ``` rsync --avz --progress --verbose /path/to/local/file username@mox.hyak.uw.edu:/path/to/remote/directory ``` Is moving files _to_ mox

3. I prepared a bisulfite genome on hummingbird (see https://github.com/RobertsLab/project-virginica-oa/blob/master/notebooks/2018-04-27-Gonad-Methylation-Bismark.ipynb). Should I move the prepared genome to Mox, or would it make sense to run the genome preparation step on Mox as well?

You can move — thanks Steven Steven B. Roberts Kenneth K. Chew Endowed Professor School of Aquatic and Fishery Sciences University of Washington, Seattle WA sr320@uw.edu - 206.866.5141 robertslab.info - @sr320

…

On Oct 12, 2018, 4:07 PM -0700, Yaamini Venkataraman ***@***.***>, wrote: I wrote a script to run bismark in Mox: https://github.com/fish546-2018/yaamini-virginica/blob/master/scripts/2018-10-12-Bismark-Revised-Parameters.sh I have a few questions/things I need clarified: 1. I assume I would have to create my SBATCH directory before I run the script 2. How do I upload the C. virginica genome and trimmed files to Mox? The file transfer instructions has information on moving files from Mox. Would I just switch the path to my remote directory and path to directory on Mox? 3. I prepared a bisulfite genome on hummingbird (see this Jupyter notebook). Should I move the prepared genome to Mox, or would it make sense to run the genome preparation step on Mox as well? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[kubu4]

In regards to No.2, the most frictionless way to transfer files is to always initiate transfers from Mox. This will allow you to transfer files between machines without having to deal with the two-factor authentication junk.

[kubu4]

So, I think yes, what you asked is correct. Here's an example of transferring files from a remote computer to Mox (this is initiated from Mox):

rsync --archive --progress --verbose username@computer_name:/path/to/remote/file /path/to/mox/directory

[yaaminiv]

I moved all my files to Mox and created my SBATCH directory. Does it matter where I store my shell script to run the job from the login node?

[sr320]

There is not really anything called an SBATCH directory.

Like class, in your user directory, I suggest creating subdirectories
like scripts, analyses, data, etc.

If input files or output files are very large or many, you would need to use scrubbed directory (deletes in 30 days).

stop by and I can show you..

see also https://github.com/RobertsLab/hyak_mox/wiki/Data-Storage-&-System-Organization

[sr320]

sbatch is the main execution command for the job scheduler (slurm). It spools up an execute node for long term or compute intensive tasks such as assemblies, blasts, or other things of that nature.

[yaaminiv]

Should clarify: SBATCH is just what's it was called in the script header. I created a system of subdirectories in the scrubbed directory for my analyses.

I just want to know where I need to save my script so I can run it. Does it matter where it is?

[sr320]

In your user directory Yes it matters

…

On Oct 17, 2018, 1:03 PM -0700, RobertsLab/resources ***@***.***>, wrote: SBATCH

[yaaminiv]

The working directory I established in my script is: /gscratch/scrubbed/yaamini/Virginica-MBD/2018-10-17-Virginica-Bismark-Revised-Parameters

I can just save the script in that working directory?

[sr320]

I am suggesting putting in your user directory (not scrubbed) But sure you can put it there.

…

On Oct 17, 2018, 1:07 PM -0700, Yaamini Venkataraman ***@***.***>, wrote: The working directory I established in my script is: /gscratch/scrubbed/yaamini/Virginica-MBD/2018-10-17-Virginica-Bismark-Revised-Parameters I can just save the script in that working directory? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[yaaminiv]

Oh, got it. I will put it in my user directory.