Repository navigation
Deploy a Collection
This tutorial will explain how to deploy a collection onto a server or onto your local machine.
You will first need secure shell access (ssh) to the server.
Create an ssh key on your local computer that you will use to access the server. Do this by entering the following commands from your terminal:
mkdir ~/.ssh
cd ~/.ssh
ssh-keygen
[accept all defaults]
cat id_rsa.pub
Send the contents of the file id_rsa.pub to your server administrator (which will be printed by the last command above). This is the public key from your key pair. Your sysadmin will log into the server and add your public key as a new line to the server-side file ~/.ssh/authorized_keys.
Ask the server administrator for the IP address and username that you should use for ssh access.
Create a file on your local machine called ~/.ssh/config and add the following contents to it:
host my_server_name
Hostname 13.43.100.30 <-- replace with your server IP address
User ubuntu <-- replace with your username on the server
IdentityFile ~/.ssh/id_rsa
You should now be able to access the server with the following command from your terminal:
ssh my_server_name
Note: You may first need to be connected your institution's VPN, depending on the firewall settings that your server administrator has implemented on the server.
If your server administrator has already configured the server for mySocialWatcher, then you can skip this step.
If you are using a new server that has not yet been configured, then you will need to do some initial work to set it up. The shell script located in ./docker/deploy.sh contains the commands that you need to execute on the server.
If the mySocialWatcher repository has not already been cloned to the server, then you will need to do this. Note that this step is also covered in the last few lines of ./docker/deploy.sh, but further explanation is provided here.
You first need to create and authorize a deploy key to allow the server access to the repository. To do this, create an ssh key on the server:
mkdir ~/.ssh
cd ~/.ssh
ssh-keygen
[accept defaults]
cat id_rsa.pub
Go to the repository settings at https://github.com/OxfordDemSci/mySocialWatcher or ask the repository administrator to complete this step for you. Click on repository "Settings" then select "Deploy Key" in the left panel. Select "Add deploy key" and paste the contents of the file ~/.ssh/id_rsa.pub from the server into the box.
Back on the server, configure ssh github validation:
echo "Host github" >> ~/.ssh/config
echo -e "\tHostName ssh.github.com" >> ~/.ssh/config
echo -e "\tUser git" >> ~/.ssh/config
echo -e "\tIdentityFile ~/.ssh/id_rsa" >> ~/.ssh/config
Now you can clone the repository onto the server:
cd ~
git clone github:OxfordDemSci/mySocialWatcher
You may need to switch branches and/or pull the most recent commits down to the server. To check which branch you are currently on:
cd ~/mySocialWatcher
git branch
To change branches:
git checkout new_branch_name
To pull the most recent commits:
git pull
The crawler will search for new data collected by your collectors and write it into the centralised datahub. If your server does not yet have the mySocialWatcher crawler running on it, then you need to complete this step.
The crawler environment file contains the URL of the datahub and an API access token that authorizes write-access to it. An example of this file is in ./config/_example_private/crawler_env. You need to create a copy of this file in ./config/private/crawler.env and fill it in with the correct information for the centralised datahub (ask your server admin if you do not know).
The GitHub repository ignores your credentials and environment files (contained in your local directory ./config/private/), so you will need to manually copy them to the server. From the terminal of your local machine, use the following commands to do that:
cd ~/[my_local_repo_path]/mySocialWatcher
scp -r ./config/private/ psw_datahub:~/mySocialWatcher/config/private/
This command will copy the entire contents of your ./config/private folder over to the server.
To estimate total run time for a collection - including collections that have multiple .json files in them – you can use the function below:
https://github.com/OxfordDemSci/mySocialWatcher/blob/dev/mysocialwatcher/collector/utils.py#L103
This function will output the total estimated run time, taking into account the specific specs, sleep time, and number of tokens.
You should have created a shell script that will help you deploy your collection. Execute this shell script on the server:
cd ~/mySocialWatcher
./docker/collectors/[collector_name]/deploy.sh
This will activate the cron scheduler to run your collection on schedule, generate all of the files defining your collection, and build the docker containers. It will not launch the collections.
If the crawler has not already been deployed on the server, execute the existing shell script to deploy the crawler:
cd ~/mySocialWatcher
./docker/crawler/deploy.sh