Computing Tools for CS studies exercises week 3: Remote access & shell scripts in Linux
Työväline 2013 exercises 3/4
Advanced Computer Tools exercises week 3: Remote access & shell scripts in Linux
It is recommended to keep task journal about final commands you used & comments about the exercises so that you remember something when instructor asks you questions about how you did the exercises. Thank you!
Linux remote access
It is recommended that you perform the remote access tasks on a netbook, laptop or your home computer. It is also possible to complete them on the workstations in the department, but it makes more sense to try them on a separate system. Nearly every single exercise involves connecting to your departmental home folder, so the exercises may seem rather pointless if you’re already logged on department's workstation.
If you don’t have access to a separate computer, and have to do the exercises on a departmental workstation, you should:
In exercises where you have to connect or send files to the server at shell.cs.helsinki.fi, you should replace this with users.cs.helsinki.fi. This server has a separate home folder, so it makes at least a little bit of sense to move files around.
In the task [ssh-keys] you should connect to the UKKO cluster.
In the task [cron] (and next one), add the cron table to shell.cs.helsinki.fi. Cron tables are per-machine, and you are not allowed to add them to the computers in PAJA.
On campus you can connect your laptop or netbook to the internet wirelessly. The easy option is the old HUPnet network, but if only to avoid being constantly asked for your password, you should connect to the newer Eduroam network instead.
- Windows, Linux, OSX
- Android (In finnish, Pictured guide)
- iOS (In finnish, Pictured guide)
In the task [ssh] we connected to the remote server at shell.cs.helsinki.fi.
Connect to the server at users.cs.helsinki.fi using ssh.
Type ls -l into the terminal, to see the directory contents. Please read the README file mentioned in the login message as well.
As you can see, your home directory on the users server is different from the one on other departmental servers (like shell or melkinpaasi). The users server is used mainly to run web applications, and you may come across it again on the Database Application Project -course.
Copy a file or a folder from your home computer or laptop to your home directory at the department. In Linux you can use the command scp, or in Windows the program pscp.exe that comes with Putty.
(You should open the Command Prompt in Windows, but you’ll have to look up instructions for the file transfer yourself. In fact, you’ll get off much lighter if you don’t use Windows in the command line exercises.)
The command: scp source_file username@shell.cs.helsinki.fi: target_directory (replace with the appropriate file and folder names, and your username on the CS dept computers).
You may leave the target location empty, but you will need the colon; it tells scp that you mean a directory on a remote server, not on the local hard drive. Type man scp into the terminal to see what the handy options -p and -r do.
Similarly, you can copy files or folders from server:
scp username@shell.cs.helsinki.fi:source_file target_location
scp -pr username@shell.cs.helsinki.fi: source_directory target_location
This time you have to give both the source and the target location. If you are copying into the directory you are currently in, you may use . (full stop) as the target location.
It is often easier to use a graphical scp-interface:
The file transfer protocol you’re using is sftp.
Install one of these (depending on your operating system) and connect to the department (shell.cs.helsinki.fi).
If you are on a department workstation and you do not have another computer in your use, you can proceed to the next exercise.
More convenient still is an ssh network drive: your departmental home directory will be displayed in your file browser as if it were an external (slow) hard drive!
In Ubuntu: Places → Connect to Server (in Unity, you’ll find “Connect to Server” in the File menu of the File Browser). You’ll need to fill in the following settings:
- Service type: "SSH"
- Server: "shell.cs.helsinki.fi"
- Folder: "/home/fs/username" (fill in your own username)
Add bookmark, Bookmark name: "department" (or similar)
OS X: MacFusion or ExpanDrive.
Windows: Dokan (not tested), or ExpanDrive.
Now that you’ve been using ssh etc, you might be getting a little annoyed at being constantly asked to type in your password. You’re not alone in this, and the solution is to have a public-private key pair.
Create a private and public key pair in the ~/.ssh directory of your local computer, by running ssh-keygen. Copy the public key to the ~/.ssh directory of your departmental home directory. Follow any tutorial you may find (or try man ssh).
Test: ssh username@shell.cs.helsinki.fi. You should no longer be asked for a password!
You need to assign a passphrase that is different from your password at the department. Why? BECAUSE OF COMPUTER SECURITY! (It is forbidden to have ssh keys without a passphrase on the department account)
Please note that if you do not choose a passphrase, it will work automatically without ever asking you for a password, but on the other hand, anyone who has access to your computer may copy the contents of your /.ssh folder and use your private ssh key to log on to your departmental account.
You may end up feeling a little silly; now, instead of asking you for your departmental password, ssh will ask for the passphrase of the key, which should be different from any other password you use. This problem is solved by ssh-agent, which remembers your passphrase until you restart your computer. Refer to the guides mentioned (the command ssh-add will add the ssh key to the agent’s memory). This is the recommended approach. As a bonus, in OS X the keychain handles everything automatically.
Life gets easier still: add shell.cs.helsinki.fi to the ssh config file. Instructions from internet or try man ssh_config.
Test: ssh department (or whichever name you chose) should work, still without asking for a password (except the first time during a session)!
In fact, ssh is not only restricted to the command-line interface, but it can be used to convey a graphical X11 environment! Works straight away in Ubuntu, as well as in OS X, as long as you install X11 (from the OS X installation disc or the more recent XQuartz). In Windows you’d have to install an X11 server, but we’ve not tested this so try it at your own risk.
Command: ssh -Y department (or ssh -Y username@shell.cs.helsinki.fi if you didn’t do the previous couple of tasks)
Try to run any graphical application, such as chromium-browser or gedit.
You can also run an entire GNOME desktop, complete with the top and bottom bars: gnome-session. Nuts or what? Warning: doing this will probably crash your local GUI if you’re running Ubuntu. It does work fine in OS X and might even be useful in case you sometimes need to run a crippled Ubuntu session.
Use rsync to copy a largish file (about 1-10 megabytes in size, an mp3-file or a full-size photo will do fine) from your local computer to the departments server.
Once you get the copying to work, run the same command again and see which time it works faster (1st or 2nd?)! By default, if rsync transfers successfully, it won’t print anything on the screen. Therefore, the option --progress below. Try the option -v as well.
rsync --progress largish_file department:rsync --progress largish_file department:
So move the SAME file again! Which one was faster (1st or 2nd)? In addition: rsync understands the “department” alias you set up in exercise [ssh-config]!
Rsync comes as standard in OS X (as well as Ubuntu), and for Windows you can get it off the Internet.
Rsync works similarly to scp which we used before to copy the files over ssh. But rsync does not copy, it synchronises: it does not copy unaltered files again, and for altered files it only copies the bits that have changed. This will save you lots of time and internet bandwidth, and is the perfect (command line) backup tool. Rsync is just insanely great!
Use rsync to copy a directory (e.g. ~/Documents) from your local computer to the directory ~/bakup on departments server. In rsync you can copy a directory by using the option -a, which you’d use pretty much always anyhow. Cf. man rsync or the best tutorial I ever found, from 2004.
What we want here is that the ~/Documents directory on your home computer/laptop is the same as the directory ~/bakup/Documents on your departmental drive, not that the contents of your Documents directory are in the directory ~/bakup at the department!
Note: rsync operates differently depending on whether you include “/” after the source directory. Usually you don’t want to use “/”, so that the directory, not only its contents, are copied.
This task is about checking and reviewing csfs home directory, to make sure it has a smart permissions setup.
- Open a terminal. Navigate to the directory "public_html" located in csfs. Run
ls-la. What you’re interested in:- "." (refers to the current directory, or "public_html")
- ".." (refers to the directory one level up, that is, the root of your home directory)
- files in your public_html directory (such as index.html)
- The following are good (safe) permissions to have:
drwx--x--x 122 sampumon tkol 12288 Aug 31 18:00 . drwx--x--x 100 sampumon tkol 4096 Sep 14 12:32 .. -rw-r--r-- 1 sampumon tkol 3091 Dec 28 2011 index.html - Cf. directions for interpreting the
ls-lacommand (the first four paragraphs). - Check which group you belong to: type in
id. uid is your username, gid your default group. For those majoring in CS this is tkol (an abbreviation of the old name of the department, "TietojenKäsittelyOpin Laitos"), for those taking CS as a minor subject it is the abbreviation of your own department. - Consider the following.
- If the directory has read permissions (r) for group (g), or other (o), other students can take a look at what files you have available.
- If files have read permissions for group/other, other students can also take a look at the files themselves.
- By default, new files will have read permissions for everyone. However, others will not see the files if the directory containing the files does not have execute permissions (x) for group/other.
- The department’s web server (Apache) belongs to the group other (o). For the web server to have access to the public_html directory, the full path to the directory (e.g. /cs/fs/home/skaipiai/public_html) must have execute permissions (x) set for other (o). This was done in the task [public_html].
- In addition, all the files in public_html should be given read permissions (r) for other (o). You can give permissions symmetrically to all users (a), which means to (ugo) (this is the default behaviour for new files you create).
- If the directory public_html is set to be read only by you, no one else (including the web server) will not see the directory listing. However, individual files that have read permissions set will be accessible, if one knows (or can guess) the filename. For example, the icon at http://www.cs.helsinki.fi/u/skaipiai/windows.svg that we used before is visible to all the internets, even though the public_html directory listing is not.
- If you have an index.html file in your public_html directory (or its subdirectories), it will automatically prevent the directory listing from being seen on the Internet, but other students at the department may still be able to view your file listing through the department’s file system if the directory has read access set for others than yourself.
The task: Set the permissions for your directories so that others can’t snoop around in them unless you specifically want to let them. Also check the permissions for the directories in your csfs home directory (such as Documents, Pictures, Downloads etc.).
In particular, the root of your home directory (~) should not have read/write permissions set for others than yourself.
Now then! All of the previous tasks have led you to this moment. The following tasks are not difficult, and people in Paja will help. If the tasks still feel difficult, remember that you don’t need to complete all of the tasks to pass this course.
If you are working on these exercises using schools computers. Do following exercises using one of deparments servers.
man <command> (on occasion, help <command>), for extra stuff try info <command>
- Shell scripting primer: http://www.vectorsite.net/tsshell.html
- Command Reference: http://www.pixelbeat.org/cmdline.html
- Shell scripting course: http://www.dartmouth.edu/~rc/classes/ksh/
- UNIX tutorial for beginners: http://www.ee.surrey.ac.uk/Teaching/Unix/
Script==shell-script==command==program==file to execute.
All tasks are run in the terminal (command line), but you can edit the files in any text editor. The files are all quite ordinary text files. You can complete these tasks on (at the least)
- your home computer/laptop/netbook if it runs Ubuntu
- the department servers over ssh.
All scripts (program files to execute) need execute permissions (chmod +x). Scripts are run by executing the following command while in the same directory as the script file
./script_file [parameters]
Once you’ve completed the task [bash-scripts] below, all your scripts (that you’ve put in the ~/scripts directory) should work in any directory, by executing command script_name [parameters].
Task: Create a command file-size, which prints the size (in bytes) of the file given as a parameter. For example: ./file-size file-size would return the size of the file containing the script file-size. The command ./file-size * would return the size of all the files in the directory combined. What should be calculated as the size of directories?
# Solution 1: for+wc+awk (the size of directories is 0)
for f in "$@"; do
s=$(wc -c "$f" 2> /dev/null | awk '{ print $1 }')
sum=$((sum+s))
done
echo for+wc+awk: $sum
# Solution 2: ls+awk+bc (calculates the size of the directories’ inode or similar)
echo -n "ls+awk+bc: "
(ls -ld "$@" | awk 'ORS="+" { print $5 }' ; echo 0) | bc
# Solution 3: du+tail+cut (calculates the sizes of directories recursively based on their contents)
echo -n "du+tail+cut: "
du -bc "$@" | tail -1 | cut -f 1You can try the example task by copying the indented 14 rows above (including the #-comments) into a file called file-size in your home directory, giving it execute permissions (chmod +x file-size), and executing the command in the terminal ./file-size file-size or ./file-size * (running the second one will take a while).
Actually, the following tasks are easier than this example was!
Create a shell script kaiku, which prints on the screen whatever parameters you give it. Like this:
skaipiai@wel-68:~$ ./kaiku today’s a grand day
today’s a grand day
Save the file in your home directory as kaiku, give the file execute permissions, and try it out.
The script needs to work with n parameters (meaning that there might be more words than four...)
Create a directory scripts in your home directory and move the kaiku-script (the file, that is) there.
Add the scripts directory to path. Meaning, add the following line to the .bashrc file in your home directory.
export PATH=$PATH:~/scripts
Close the current terminal and open a new one. In the future, you should save all your scripts in the scripts directory, so that they will work in the terminal at all times (no matter what directory you’re in) and without ./ at the beginning. Try it out:
skaipiai@wel-68:~$ kaiku getting better and better\!
getting better and better!If you are working on departments servers, you need to add following line to you ~/.profile file
. "$HOME/.bashrc"
Create a shell script wget-images, which uses the command wget to get all the images (that is, at least the jpg- and png-files) from the www-address you’ve given as a parameter.
Save the file in the scripts folder you created in the previous exercise and give the file execute permissions.
Try your script out, for example on the Spring snow gallery. Execute command:
wget-images www.cs.helsinki.fi/u/skaipiai/kevätlumi/
OBS. Added a slash "/" at the end of URL above, makes task easier :)
The directory you’re currently in should now contain the files etu.jpg and sivu.jpg, but nothing else (except at the most the file robots.txt), and especially no subdirectories whatsoever.
These first two shell-scripts could just as easily have been made as aliases, because 1) they are short (one line in length) and 2) their variables (which the user/you gave as parameters, such as $@) are at the end of the command. In the next tasks the variables will be placed between the commands and the parameters (twice, even), so a simple alias will not do any longer.
Create a script manpdf, similar to the one in the task [alias], which’ll make a pdf-file out of a given man-page. E.g. "man bash" → bash.pdf:
man -t bash | ps2pdf - bash.pdf
Save your file in the scripts directory, give it execute permissions and try it out.
In this exercise you shall write your own cron. Crontabs are used to automate the operating system (in this case Linux) to run scripts and commands at a given time regularly.
You should not start doing anything before you have read the whole exercise and the notes.
Commands! The command to edit the crontab is crontab -e but it is quite dangerous since e and r are located next to each other on the keyboard and crontab -r deletes your crontab (and there is no UNDO!)
Log file: You should write a script for the cron that appends (adds to the end) the timestamp and the text “Cron done” into a file, for example ~/bakup.log.
Recommended workflow: create your own crontab file, for example "~/kron.tab". You should add a row to your kron.tab that executes your Log file script once a minute. It comes into use with the command crontab ~/kron.tab. If it prints nothing, it went through! Check the cron manual from the internet! You can see the current crontab with the commant crontab -l.
NOTE. Crontabs are per-computer. If you are doing this exercise with your home computer or laptop, everything is fine. BUT if you are using department’s workstations you should take into account that the crontab is installed on the workstation you are logged into (the computer itself). If you change workstations, crontab is empty again AND your previous crontab is still running on the previous workstation. If you do not have another computer to use, put your crontab to the shell.cs.helsinki.fi!
When the instructor is present and you have showed him/her your kron.tab, you should now delete your silly once a minute cron. If you want another cron for your personal use, you can do that later.
Create an image-manipulation script thumb, which’ll create a different-sized version of an image file given as a parameter, using the convert-command in ImageMagick.
OBS. This task is quite a bit harder than the previous ones.
An example of how the final thumb-command should run (executed in the directory into which you retrieved images with your wget-images script):
$ ls
etu.jpg sivu.jpg
$ thumb 600x *.jpg
$ ls -l
-rw-r--r-- 1 samuli staff 462321 4 Tou 10:17 etu.jpg
-rw-r--r-- 1 samuli staff 409224 4 Tou 10:25 sivu.jpg
-rw-r--r-- 1 samuli staff 147815 2 Lok 23:23 etu-600x.jpg
-rw-r--r-- 1 samuli staff 117912 2 Lok 23:23 sivu-600x.jpg
$ identify *.jpg
etu.jpg JPEG 1200x800 1200x800+0+0 8-bit DirectClass 451KiB …
sivu.jpg JPEG 1200x800 1200x800+0+0 8-bit DirectClass 400KiB …
etu-600x.jpg JPEG 600x400 600x400+0+0 8-bit DirectClass 143KiB …
sivu-600x.jpg JPEG 600x400 600x400+0+0 8-bit DirectClass 115KiB ...Don’t worry, ImageMagick will take care of everything, all you need to worry about are the file names!
It is best to create this script in sections. For example, the first version can only convert one image file. The second can convert many. The third can move the “600x” (or similar) bit to the end of the file name so that the alphabetical order remains.The fourth version will save the new image files into the current directory and not in their original directory (which is the ImageMagick default setting).
The script must have at least two (2) of the following four (4) features. You may implement these features in any order you like.
- thumb 300x megatron.jpg → 300x-megatron.jpg
- thumb 300x *.jpg → {300x-megaman.jpg, 300x-megatron.jpg}
- thumb 300x *.jpg → {megaman-300x.jpg, megatron-300x.jpg}
- thumb 300x orig/*.jpg → {./megaman-300x.jpg, ./megatron-300x.jpg}
Tips and ready-made solutions:
-
If your Ubuntu system doesn’t have the convert-function, install ImageMagick: "sudo apt-get install imagemagick".
-
The command shift moves command line parameters to the left; $1 disappears, $2→$3, $3→$2 etc. $@ still comprises all parameters except the original $1.
-
Convert is not able to deal with multiple source and target files very well, in a human-friendly way, so you might want to create a separate for-loop where you process the image files one by one
-
The following will be useful when you’re constructing the name of the target file:
ext=${f##*.} base=$(basename $f .$ext) dir=$(dirname $f) -
Try out what these variables contain. Add the following line to your thumb-script
echo dir=$dir, base=$base, ext=$extand run the script with an image file (from a different directory) as a parameter.
Alrighty! This script will hopefully come in handy in the future, for example when you’re adding thumbnails to your HTML-page.
THAT’S IT, TOODLE-PIP.
Do you want more? You’re most welcome to take the tough Linux Fundamentals -course in the second period. It’s all about BASH-scripting. But watch out, they’re real serious about this stuff there!