Skip to content

Week 03 (January 22)

Jump to bottom
Donna Calhoun edited this page Feb 8, 2018 · 23 revisions

Tuesday

  • Using %%timeit magic command to time commands in loops. See this notebook. Onen this file as a "Raw" file, and use your browser "save as" command to save this page as a .ipynb file.

  • Use Pandas to present tables.

  • Use Pipes to communicate between processes. Pipes are dedicated communication channels between two processes. The Queue object, for example, is built upon pipes.

See the demos for this week to see samples of how to use Pandas and %%timeit.

Virtual Machine

OIT has set up a virtual machine for us to use throughout the semester.

Logging into the virtual machine.

If you are on campus, you can log in to the machine through a terminal window using your BSU username and password. For example, you can log in as

$ ssh yourusername@ME471    ## e.g. donnacalhoun@ME471

If you have problems with the domain name, you can also try:

@ME471.boisestate.edu
@me471-lxt-01.boisestate.edu

If you are off-campus, you can log in using a VPN.

Once you are logged in, you will want to set up (or add to) a .bash_profile file that contains the following :

# file : .bash_profile
# ......   
# Add these lines. 
export ANACONDA=/usr/local/anaconda3
PATH=$ANACONDA/bin:./:$PATH
export PATH

You can create or edit this file using vi or nano, or you can create it locally, and copy it to your account on the virtual machine using scp (see below). This file should be stored in your home directory on the virtual machine.

After you have edited this file, run it at the command line, and check that you have set up your path correctly :

$ source .bash_profile
$ which python

If the result is /usr/local/anaconda3/bin/python, you are good to go. Next time you log in, this file will be run automatically, so will always access the correct Python executable.

Running Jupyter notebooks remotely

In what follows, the remote machine refers to the virtual machine supplied by OIT, although the following instructions will work on any similarly-configured remote machine. The local machine is your desktop or laptop machine.

Jupyter notebooks can be created, and edited on a local machine, but stored and run on a remote machine. To connect your local browser (where a remote notebook will be edited), use the following commands:

  1. Log into the remote machine and issue the following command at the bash command prompt:

    (remote) $ jupyter notebook --no-browser --port=8100

where the port number 8100 is replaced by your number. Please use a port number ending in the number you were assigned in class.

  1. On the local machine, you will issue the command :

    (local) $ ssh -N -f -L 127.0.0.1:8100:127.0.0.1:8100 donnacalhoun@me471.boisestate.edu

where you will replace 8100 with your port number, and replace donnacalhoun with your user name.

  1. In local browser, navigate to the URL

    (local) $ http://127.0.0.1:8100/

again, replacing the port number with your port value.

You can now create, edit, and run Jupyter notebooks that are stored on the remote machine.

Copy local notebooks to remote machine

To copy locally stored notebooks to the remote machine, do the following

(remote) $ cd
(remote) $ mkdir notebooks       # Or use another directory name

Then on the local machine

(local) $ # cd to location of local notebooks
(local) $ scp my_notebook.ipynb yourusername@me471.boisestate.edu:notebooks

Note : Be sure to use the : in specifying the copy location.

Generate a public/private ssh key.

If you are tired of typing your password each time you log in, you can generate an ssh public/private key pair.

On your local machine :

(local) $ cd
(local) $ mkdir .ssh     # Ignore errors if you already have a directory .ssh
(local) $ cd .ssh
(local) $ ssh-keygen -t rsa
(local) $ # Hit enter several times until the bash prompt returns
(local) $ scp id_rsa.pub yourusername@me471.boisestate.edu:      # don't forget colon at end

Log into your remote machine (this is the last time you should need your password)

(remote) $ mkdir .ssh    # Ignore errors if you already have this directory
(remote) $ mv id_rsa.pub .ssh
(remote) $ cd .ssh
(remote) $ cat id_rsa.pub >> authorized_keys

Exit the remote machine and log in again. You should be let in immediately without the password prompt.

Stack exchange questions.

Here are two stack exchange questions I have posted recently : Both are about communication with the child-process.

Communication overhead

Communication using pipes

Thursday

Here is a link to an algorithm on finding the median (the "median of medians" algorithm)

And here is a pre-print of the same paper, but with a more complete description of the algorithm above.

@DonnaCalhoun Introduction to Parallel Computing (ME471/571, BSU)

Clone this wiki locally