Uploading data to NCBI SRA

[yaaminiv]

@kubu4 How do you usually upload raw data to NCBI SRA? I'm trying to upload the C. gigas Manchester files through the Aspera plug-in and it keeps timing out.

[kubu4]

I always use FTP.

[kubu4]

Since you'll likely want to use your laptop and take it places with you, I'd initiate the FTP transfer from one of the Roberts Lab computers using screen. screen allows you to kill a remote connection, but the processes running on the computer keep going. So, the process would be:

1. SSH into a Roberts Lab computer.
2. Mount Owl (if you haven't already) somewhere.
3. Start screen: screen
4. Navigate to desired directory on Owl.
5. Initiate FTP to NCBI.

Then, you can exit the connection to the Roberts Lab computer any time and SSH back in and check in on your screen session at any time.

[yaaminiv]

Mount Owl (if you haven't already) somewhere.

Can I have owl mounted on my personal laptop, or does it need to be on a Roberts Lab computer?

[kubu4]

I think you can mount it on your laptop, but you won't be able to shut your laptop down or disconnect it from the network during the FTP transfer. That's why I recommended SSH-ing to a Roberts Lab computer to do this.

[yaaminiv]

Got it! And how do I mount owl on a Roberts Lab computer I'm SSH-ing into...

[kubu4]

Use this example:

sudo mount --verbose -t cifs //owl.fish.washington.edu/web /mnt/owl/ -o credentials=/home/<user>/.owl_creds,vers=1.0

Here's what's happening:

• sudo mount --verbose: Telling computer to mount a drive in verbose mode. You may not need to use sudo - see below.

• -t cifs: File system mount type.

• //owl.fish.washington.edu/web/nightingales: Location on Owl that you want to mount.

• /mnt/owl/: Location of empty folder on the computer you're logged into. Preferably, you should create a dedicated directory in your /home/ directory. That should allow you to run the mount command without sudo.

• -o credentials=/home/<user>/.owl_creds: Path to a "hidden" credentials file. Contains username and password info for your Owl login. Formatted like this:

     username=<username>
     password=<password>

  IMPORTANT: This method isn't very secure. If you use this method, be sure to change the permissions on that credentials file:

  • chmod 700 .owl_creds

• ,vers=1.0: Specifies SMB version to use, since Owl is using an older version.

[yaaminiv]

/mnt/owl/: Location of empty folder on the computer you're logged into. Preferably, you should create a dedicated directory in your /home/ directory. That should allow you to run the mount command without sudo.

So I create a folder (ex. owl) in my home directory, then change /mnt/owl/ to that path?

This method isn't very secure. If you use this method, be sure to change the permissions on that credentials file

Is there a way to log in without using this method?

[yaaminiv]

I'm SSH-ed into ostrich and tried mount --verbose -t cifs //owl.fish.washington.edu/web /owl/ -o username=<yaaminiv>,vers=1.0:

-bash: yaaminiv: No such file or directory

Similar error with sudo mount --verbose -t cifs //owl.fish.washington.edu/web /owl/ -o credentials=/home/<user>/.owl_creds,vers=1.0

-bash: user: No such file or directory

[yaaminiv]

WHOOPS hang on did this:

sudo mount --verbose -t cifs //owl.fish.washington.edu/web /owl/ -o credentials=/home/yaaminiv/.owl_creds,vers=1.0

and now have the following error: srlab is not in the sudoers file. This incident will be reported.

[yaaminiv]

Tried running without sudo:

mount --verbose -t cifs //owl.fish.washington.edu/web owl/ -o credentials=/home/yaaminiv/.owl_creds,vers=1.0

Got this error:

mount: illegal option -- -
usage: mount [-dfruvw] [-o options] [-t external_type] special node
       mount [-adfruvw] [-t external_type]
       mount [-dfruvw] special | node

[kubu4]

My instructions were for Linux, not a Mac; sorry about that. Not entirely sure what to do here. However, the error is related to the use of --verbose. Get rid of that and see what happens...

[yaaminiv]

ostrich:~ srlab$ mount -t cifs //owl.fish.washington.edu/web owl/ -o credentials=/home/yaaminiv/.owl_creds,vers=1.0
usage: mount [-dfruvw] [-o options] [-t external_type] special node
       mount [-adfruvw] [-t external_type]
       mount [-dfruvw] special | node
ostrich:~ srlab$ sudo mount -t cifs //owl.fish.washington.edu/web /owl/ -o credentials=/home/yaaminiv/.owl_creds,vers=1.0
Password:
srlab is not in the sudoers file.  This incident will be reported.

My instructions were for Linux

Tried SSH into roadrunner:

sudo mount -t cifs //owl.fish.washington.edu/web owl/ -o username=yaaminiv,vers=1.0

srlab@roadrunner:~$ cd owl
srlab@roadrunner:~/owl$ ls
Athaliana    index.html                        phainopepla   trochophora
dlf          index.php                         porites       web_images
eimd         kaitlyn                           sam           wetgenes
fish546      lilmax                            scaphapoda    whale
generosa     nightingales                      spartina
halfshell    O_lurida_genome_assemblies_BGI    symbiodinium
hesperornis  P_generosa_genome_assemblies_BGI  testing

SUCCESS (so far). I'll try using screen now that I have owl mounted!

[yaaminiv]

Once I mounted owl on roadrunner, I was able to use screen and ftp -i!

I exited my terminal window and logged back in, but I can't find my screen session...I logged back into roadrunner via SSH, navigated to my owl directory, and started screen, but all I see is a blank Terminal window. How do I check on my session?

[kubu4]

Exit the current screen session:

exit

Should print out something like:

[screen is terminating]

Then, list active screen sessions:

screen -ls

Attach the screen session you want:

screen -r <number>

The number will be shown at the beginning of .pts-11.roadrunner (02/15/2022 01:50:14 PM) (Attached) when you run the screen -ls command.

[sr320]

omg this is exciting 🥰 - will she get in or not? This could definitely be a movie trailer 😆 I feel like there is a bomb countdown ...

[kubu4]

LOL!

[yaaminiv]

Thanks for the screen tutorial (that I totally could have googled myself whoops)! FTP is still going.