Motu is a high efficient and robust Web Server which fills the gap between heterogeneous data providers to end users. Motu handles, extracts and transforms oceanographic huge volumes of data without performance collapse. This client enables to extract and download data through a python command line.
Python HTML Roff Shell
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
.settings
assembly
src
.gitignore
.project
.pydevproject
LICENSE
MANIFEST.in
Motu-client-python - CONSOLE - DESCRIBE PRODUCT - CAS SSO QT.launch
Motu-client-python - CONSOLE - DOWNLOAD - CAS SSO QT.launch
Motu-client-python - CONSOLE - GET SIZE - CAS SSO QT.launch
Motu-client-python - DOWNLOAD - No CAS SSO.launch
Motu-client-python - HELP.launch
README.md
patchPOMtoBuild.sh
pom.xml
setup.cfg
setup.py

README.md

Motu Client Python Project

@author Product owner tjolibois@cls.fr
@author Scrum master, software architect smarty@cls.fr
@author Quality assurance, continuous integration manager smarty@cls.fr

How to read this file? Use a markdown reader: plugins chrome exists (Once installed in Chrome, open URL chrome://extensions/, and check "Markdown Preview"/Authorise access to file URL.), or for firefox (anchor tags do not work) and also plugin for notepadd++.

Be careful: Markdown format has issue while rendering underscore "_" character which can lead to bad variable name or path.

Summary

Overview

Motu client "motu-client-python" is a python script used to connect to Motu HTTP server in order to:

  • extract the data of a dataset, with geospatial, temporal and variable criterias (default option)
  • get the size of an extraction with geospatial, temporal and variable criterias
  • get information about a dataset

This program can be integrated into a processing chain in order to automate the downloading of products via the Motu.

Build

From the root folder runs the command:

./patchPOMtoBuild.sh  
mvn clean install -Dmaven.test.skip=true
[...]
[INFO] BUILD SUCCESS
[...]

This creates two archives in the target folder:

  • motu-client-python-$version-$buildTimestamp-src.tar.gz: Archive containing all the source code
  • motu-client-python-$version-$buildTimestamp-bin.tar.gz: Archive ready to be installed

Installation

You must use python version 2.7.X or later.
This program is not compatible with Python 3.X versions.
There is two methods to install the client, by using PIP or from a tar.gz file.

Using PIP

Python Package Index is used to ease installation.
If your host needs a PROXY set it, for example:

export HTTPS_PROXY=http://myCompanyProxy:8080  

Then run:

pip install motu-client  

Now "Motu-client" is installed, you can configured it and use it.

From tar.gz file

Deploy the archive (file motu-client-python-$version-bin.tar.gz available from GitHub release) in the directory of your choice.

tar xvzf motu-client-python-$version-$buildTimestamp-bin.tar.gz

Create a configuration file and set the user and password to use to connect to the CAS server.

Configuration

All parameters can be defined as command line options or can be written in a configuration file. The configuration file is a .ini file. This file is located in the following directory:

  • on Unix platforms: $HOME/motu-client/motu-client-python.ini
  • on Windows platforms: %USERPROFILE%\motu-client\motu-client-python.ini

The expected structure of file is:

[Main]  
user=john  
pwd=secret  
log_level=10  
proxy_server=proxy.domain.net:8080  
proxy_user=john  
proxy_pwd=secret  
motu=http://motu-ip-server:port/motu-web/Motu
product_id=dataset-psy2v3-pgs-med-myocean-bestestimate  
date_min=2010-11-08 12:00:00  
date_max=2010-11-10  
latitude_min=-75.0  
latitude_max=30.0  
longitude_min=20.0  
longitude_max=120.0  
depth_min=  
depth_max=  
variable=  
out_dir=./out_dir  
out_name=test.nc  
block_size=65535  
socket_timeout=  

Usage

Starts the motu python client.

Usage from PIP installation

python -m motu-client -h  
python -m motu-client [options]

Options are listed below.
Method to used when it has been installed with PIP method.

Usage from tar.gz installation

./motu-client.py  -h  
motu-client.py [options]

Method to used when it has been installed with tar.gz method.
Usefull if host is offline and has no Internet access.

Options:

  • -h, --help show this help message and exit

  • -q, --quiet print logs with level WARN in stdout, used to prevent any output in stdout

  • --noisy print logs with level TRACE in stdout

  • --verbose print logs with level DEBUG in stdout

  • --version show program's version number and exit

  • --proxy-server=PROXY_SERVER Proxy server (url) used to contact Motu

  • --proxy-user=PROXY_USER Proxy user name (string)

  • --proxy-pwd=PROXY_PWD Proxy password (string)

  • --auth-mode=AUTH_MODE the authentication mode: [default: cas]

    • none for no authentication
    • basic for basic authentication
    • cas for Central Authentication Service
  • -u USER, --user=USER User name (string) for the specified authentication mode

  • -p PWD, --pwd=PWD the user password (string) for the specified authentication mode. Special characters can be used.

    • Example 1 From a Windows batch, if your password contains a percent character, double the percent character: If password is CMS2017@%! then enter -u username-p CMS2017@%%!
    • Example 2 From a Windows batch, if your password contains a space character, set password between double quotes: If password is CMS2017 @%! then enter -u username-p "CMS2017 @%%!"
    • Example 3 From a Linux shell, if your password contains a space character, set password between simple quotes: If password is CMS2017 @%! then enter -u username-p 'CMS2017 @%!'
  • -m MOTU, --motu=MOTU Motu server url, e.g. "-m http://localhost:8080/motu-web/Motu"

  • -s SERVICE_ID, --service-id=SERVICE_ID The service identifier (string), e.g. -s Mercator_Ocean_Model_Global-TDS

  • -d PRODUCT_ID, --product-id=PRODUCT_ID The product (data set) to download (string), e.g. -d dataset-mercator-psy4v3-gl12-bestestimate-uv

  • -t DATE_MIN, --date-min=DATE_MIN The min date with optional hour resolution (string following format YYYY-MM-DD [HH:MM:SS]), e.g. -t "2016-06-10" or -t "2016-06-10 12:00:00". Be careful to not forget double quotes around the date.

  • -T DATE_MAX, --date-max=DATE_MAX The max date with optional hour resolution (string following format YYYY-MM-DD [HH:MM:SS ]), e.g. -T "2016-06-11" or -T "2016-06-10 12:00:00". Be careful to not forget double quotes around the date.

  • -y LATITUDE_MIN, --latitude-min=LATITUDE_MIN The min latitude (float in the interval [-90 ; 90 ]), e.g. -y -80.5

  • -Y LATITUDE_MAX, --latitude-max=LATITUDE_MAX The max latitude (float in the interval [-90 ; 90 ]), e.g. -Y 80.5

  • -x LONGITUDE_MIN, --longitude-min=LONGITUDE_MIN The min longitude (float in the interval [-180 ; 180 ]), e.g. -x -180

  • -X LONGITUDE_MAX, --longitude-max=LONGITUDE_MAX The max longitude (float in the interval [-180 ; 180 ]), e.g. -X 35.5

  • -z DEPTH_MIN, --depth-min=DEPTH_MIN The min depth (float in the interval [0 ; 2e31 ] or string 'Surface'), e.g. -z 0.49

  • -Z DEPTH_MAX, --depth-max=DEPTH_MAX The max depth (float in the interval [0 ; 2e31 ] or string 'Surface'), e.g. -Z 0.50

  • -v VARIABLE, --variable=VARIABLE The variable (list of strings), e.g. -v salinity -v sst

  • -S, --sync-mode Sets the download mode to synchronous (not recommended). If this parameter is set, Motu server is called with parameter console. Otherwise , Motu server is called with parameter status.

  • -o OUT_DIR, --out-dir=OUT_DIR The output dir where result (download file) is written (string). If it starts with "console", behaviour is the same as with --console-mode.

  • -f OUT_NAME, --out-name=OUT_NAME The output file name (string)

  • --console-mode Write result on stdout. In case of an extraction, write the nc file http URL where extraction result can be downloaded. In case of a getSize or a describeProduct request, display the XML result.

  • -D, --describe-product Get all updated information on a dataset. Output is in XML format, API details

  • --size Get the size of an extraction. Output is in XML format, API details

  • --block-size=BLOCK_SIZE The block used to download file (integer expressing bytes)

  • --socket-timeout=SOCKET_TIMEOUT Set a timeout on blocking socket operations (float expressing seconds)

  • --user-agent=USER_AGENT Set the identification string (user-agent) for HTTP requests. By default this value is 'Python-urllib/x.x' (where x.x is the version of the python interpreter)

Usage examples

In the following examples, variable ${MOTU_USER} and ${MOTU_PASSWORD} are user name and user password used to connect to the CAS server for single sign on.
${MOTU_SERVER_URL} is the URL on the MOTU HTTP(s) server. For example http://localhost:8080/motu-web/Motu.
Commands "./motu-client.py" has to be replaced by "python -m motu-client" if it has been installed with PIP method.

Download

Download and save extracted file on the local machine

This command writes the extraction result data in file: /data/test.nc

./motu-client.py --verbose --auth-mode=none -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o /data -f test.nc

Display on stdout the HTTP(s) URL of the NC file available on the Motu server

The HTTP(s) URL is displayed on stdout. This URL is a direct link to the file which is available to be downloaded.

./motu-client.py --quiet --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o console

GetSize

See https://github.com/clstoulouse/motu#ClientAPI_GetSize for more details about XML result.

Download and save the XML file which contains the size on the local machine

./motu-client.py --size --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o /data -f getSizeResult.xml

Display the size XML result on stdout

./motu-client.py --quiet --size --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o console

DescribeProduct

See https://github.com/clstoulouse/motu#describe-product for more details about XML result.

Download and save the XML file which contains the size on the local machine

./motu-client.py -D --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o /data -f describeProductResult.xml

Display the size XML result on stdout

./motu-client.py --quiet -D --auth-mode=cas -u ${MOTU_USER} -p ${MOTU_PASSWORD}  -m ${MOTU_SERVER_URL} -s HR_MOD_NCSS-TDS -d HR_MOD -z 0.49 -Z 0.50 -x -70 -X 25 -y -75 -Y 10 -t "2016-06-10" -T "2016-06-11" -v salinity -o console

Licence

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.

Troubleshooting

Unable to download the latest version watched on GitHub from PIP

Example:

pip install motu-client  
Collecting motu-client  
  Using cached https://test-files.pythonhosted.org/packages/4a/7d/41c3bdd973baf119371493c193248349c9b7107477ebf343f3889cabbf48/motu-client-X.Y.Z.zip  
Installing collected packages: motu-client  
  Running setup.py install for motu-client ... done  
Successfully installed motu-client-X.Y.Z  

Clear your PIP cache: On Windows, delete the folder %HOMEPATH%/pip. On archlinux pip cache is located at ~/.cache/pip. After re run the command:

pip install motu-client  
Collecting motu-client  
  Using https://test-files.pythonhosted.org/packages/4a/7d/41c3bdd973baf119371493c193248349c9b7107477ebf343f3889cabbf48/motu-client-X.Y.Z.zip  
Installing collected packages: motu-client  
  Running setup.py install for motu-client ... done  
Successfully installed motu-client-X.Y.Z  

From Windows, Parameter error

From Windows, the command "motu-client.py --version" returns an error.
10:44:24 [ERROR] Execution failed: [Excp 13] User (option 'user') is mandatory when 'cas' authentication is set. Please provide it.

Analyse:
This issue comes from the fact that Windows command line does not pass parameters to python command.

Solution:

Edit the Windows Registry Key "HKEY_CLASSES_ROOT\py_auto_file\shell\open\command" and append at the end of the value %*  
Exemple: "C:\dvltSoftware\python\Python27\python.exe" "%1" %*