-
Notifications
You must be signed in to change notification settings - Fork 1
How to Move Files
If you prefer a GUI tool, you can use any number of FTP client apps to transfer files from your machine to Wynton:
The FTP connection hostname would be a Wynton transfer node: dt1.wynton.ucsf.edu or dt2.wynton.ucsf.edu.
Note: Gladstone user have additional options for high-speed data transfers to/from Gladstone, local and Dropbox locations: See internal confluence docs.
By command line, you can transfer files from any machine to your Wynton home directory:
rsync filename.tsv username@dt2.wynton.ucsf.edu:~/
If you set the sharing permissions on a file (or folder) in Dropbox to allow anyone to view via a link, then copy the link and use wget from Wynton to get a local copy of the file:
wget "https://www.dropbox.com/s/sdfd89f0a8f9af8/my_file.tsv"
Note: You may want to remove ?dl=0 from the end of the link to preserve the output filename or use the -o parameter (see next examples).
If you are downloading a directory, note that Dropbox will automatically send it as a .zip archive. Here are some examples of downloading and unzipping a directory.wget:
wget "https://www.dropbox.com/sh/sdfd89f0a8f9af8/aasd0f98sda0f8d9f8sd?dl=1" -o my_folder.zip && unzip my_folder.zip -d my_folder
Using curl, you need to specify -L to follow redirects:
curl -L "https://www.dropbox.com/sh/sdfd89f0a8f9af8/aasd0f98sda0f8d9f8sd?dl=1" -o my_folder.zip && unzip my_folder.zip -d my_folder
Note: When downloading a directory, there may be a size limit ~1GB. To get around that, you can manually create a zip or tar.gz archive of the directory and download that as a single file. In some cases, it may help to change dl=0 to dl=1 in the share link before downloading.
For files of size less than 100K, wget works on drive links following the transformation...
From: https://drive.google.com/file/d/123ABC/view?usp=sharing
To: https://drive.google.com/uc?export=download&id=123ABC
If you are copying files from an SFTP server, e.g., sequencing files from UCSF's CAT Core, log in to a Wynton data transfer node. Then, login to sftp as follows:
sftp username@fastq.ucsf.edu
Next, navigate to the directory where the files you want to copy are located. Finally, to copy them to your current directory on Wynton login node (the location from which you logged in to sftp), use the following command.
get -r *
If you are copying files that will only be needed temporarily, for example as input to a job, then you have the option of copying them directly to a global scratch space at /wynton/scratch/. There is 492TiB of space available for this purpose. It is automatically cleared after 2 weeks, but you should go ahead and delete the files when you no longer need them.
Note: it is good practice to first create your own subdirectory here and copy to that
mkdir /wynton/scracth/my_own_space
scp filename.tsv username@dt2.wynton.ucsf.edu:/wynton/scratch/my_own_space
See documentation for more about local and global scratch spaces.