Table of Contents
Many data processing pipelines need to run software that only runs on a specific combination of OS type, version, language, and library. These software could be impractical to replace or modify but need to be run side-by-side with software of incompatible environments/formats to form an integral data processing pipeline, each software being a "node" to perform a processing task. Docker containers are often the perfect solution to run software with incompatible dependencies.
PNS installed on a Docker container or a normal server allows such processing tasks to run in the PNS memory space, in a daemon process, or as an OS process receiving input and delivering output through a 'delivery man' protocol.
This Web API Server for a data processing pipeline/network node provides interfaces to configure the data processing task software (PTS) in a processing node, to make a run request, to deliver necessary input data, and to read results, all via web APIs.
The following commands are run from the fdi directory from installation.
When running Flask server, the host IP is 127.0.0.1
and port number 5000
by default. They are configurable in pnsconfig.py
. Default configuration can be overridden by ~/.config/pnslocal.py
. Copy pnsconfig.py
to ~/.config/pnslocal.py
cp fdi/pns/config.py ~/.config/pnslocal.py
and customize ~/.config/pnslocal.py
.
When in developement mode, select dev
to run local server. The serveruser
should be the name of the user of web server, usually your username if you run make runserver
. This is the default if dev
is selected.
For production deployment choose servertest
or external
. Set serveruser
depending which web server (e.g. 'apache'
).
The ptsuser
is usually the user required by the processing software. It is set to serveruser
by default. ptsuser
must have write previlige to read and write inputdir
and outputdir
, which are owned by serveruser
with mode o0775
.
On the server side (or on your computer which can be both the server and the client) edit Makefile
by changing the value of varible PNSDIR
in Makefile
the pnshome directory if you do not want the default (~/pns
).
Then run the deployment command
make installpns
to create the pns home directory and copy the demo PTS script set.
Edit ~/.config/pnslocal.py
if needed. Then
python3 fdi/pns/runflaskserver.py --username=<username> --password=<password> [--ip=<host ip>] [--port=<port>]
Contents in []
, like [--ip=<host ip>] [--port=<port>]
above, are optional.
<>
means you need to substitute with actual information (for example --port=<port>
becomes --port=5000
).
Or you can run
python3 fdi/pns/runflaskserver.py -u <username> -p <password> [-i <host ip>] [-o <port>]
in debugging mode:
python3 fdi/pns/runflaskserver.py --username=foo --password=bar -v
or just
make runserver
to use the defaults.
Do not run debugging mode for production use.
The username and password are used when making run requests.
Note
The logging level of the server is set in the config file. The -v
switch to runflaskserver
used above will set the level to logging.DEBUG
. Packages requests,
filelock, and
urllib3are fixed to
logging.WARN``.
To run all tests in one go:
make test3 [T='-u <username> -p <password> [-i <host ip>] [-o <port>] [options]']
Tests can be done step-by-step to pin-point possible problems:
Run this on the server host to verify that internal essential functions of the server work with current configuration. This runs without needing starting the server:
make test4
In ~/.config/pnslocal.py
(see above for installation and customization), set dev=True
and make sure the IP is local (127.0.0.1
). Start the server fresh in one terminal (see above) and in another terminal (on the server host) run the following:
2a: test GET initPTS script to see if reading the init script back works:
make test3 T='getinit'
2b: test PUT initialization test:
make test3 T='-k putinittest'
2c1: If the test passes, you can Run all tests in one go:
make test3
2c2: Or keep on individual tests...
test POST In-server processing
make test3 T='-k _post'
test POST PTS processing
make test3 T='-k _run'
test DELETE Clean-up the server by removing the input and output dirs
make test3 T='-k deleteclean'
Now is a good time to ...
Suppose the server address and port are 127.0.0.1
and 5000
, respectively:
Run the Flask server in a terminal (see above) and open this in a browser. The up-to-date URL is displayed in the server stating message:
An online API documentation page similar to below is shown.
{
"APIs": {
"DELETE": [
{
"URL": "http://127.0.0.1:5000/v0.6/clean",
"description": " Removing traces of past runnings the Processing Task Software.\n "
}
],
"GET": [
{
"URL": "http://127.0.0.1:5000/v0.6/init",
"description": "the initPTS file"
},
{
"URL": "http://127.0.0.1:5000/v0.6/config",
"description": "the configPTS file"
},
{
"URL": "http://127.0.0.1:5000/v0.6/run",
"description": "the file running PTS"
},
{
"URL": "http://127.0.0.1:5000/v0.6/clean",
"description": "the cleanPTS file"
},
{
"URL": "http://127.0.0.1:5000/v0.6/input",
"description": " returns names and contents of all files in the dir, 'None' if dir not existing. "
},
{
"URL": "http://127.0.0.1:5000/v0.6/output",
"description": " returns names and contents of all files in the dir, 'None' if dir not existing. "
},
{
"URL": "http://127.0.0.1:5000/v0.6/pnsconfig",
"description": "PNS configuration"
}
],
"POST": [
{
"URL": "http://127.0.0.1:5000/v0.6/calc",
"description": " generates result product directly using data on PNS.\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/testcalc",
"description": " generate post test product.\n put the 1st input (see maketestdata in test_all.py)\n parameter to metadata\n and 2nd to the product's dataset\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/echo",
"description": "Echo"
},
{
"URL": "http://127.0.0.1:5000/v0.6/run",
"description": " Generates a product by running script defined in the config under 'run'. Execution on the server host is in the pnshome directory and run result and status are returned.\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/testrun",
"description": " Run 'runPTS' for testing, and as an example.\n "
}
],
"PUT": [
{
"URL": "http://127.0.0.1:5000/v0.6/init",
"description": " Initialize the Processing Task Software by running the init script defined in the config. Execution on the server host is in the pnshome directory and run result and status are returned. If input/output directories cannot be created with serveruser as owner, Error401 will be given.\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/config",
"description": " Configure the Processing Task Software by running the config script. Ref init PTS.\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/pnsconf",
"description": " Configure the PNS itself by replacing the pnsconfig var\n "
},
{
"URL": "http://127.0.0.1:5000/v0.6/inittest",
"description": " Renames the 'init' 'config' 'run' 'clean' scripts to \"*.save\" and points it to the '.ori' scripts.\n "
}
]
},
"timestamp": 1566130779.0208821
}
Continue with tests...
Install pns on a remote host, configure IP and port, then run the tests above. This proves that the server and the client have connection and fire wall configured correctly.
Set dev=False in ~/.config/pnslocal.py (see above) and set the IP and port. Suppose the server is on CentOS. Edit pns/resources/pns.conf according to local setup, then
cp pns/resources/pns.conf /etc/httpd/conf.d
systemctl restart httpd
systemctl status http -l
then run the above with correct IP and port (edit ~/.config/pnslocal.py or specifying in command line). Start the server and run all the tests:
make test3
To run a PTS shell script instead of the 'hello' demo, change the `run
parameter in the config file, e.g. to run the script named runPTS.vvpp`
run=[join(h, 'runPTS.vvpp'), ''],
restart the server. run
make test4
TBW
400
{'error': 'Bad request.', 'timestamp': ts}
401
{'error': 'Unauthorized. Authentication needed to modify.', 'timestamp': ts}
404
{'error': 'Not found.', 'timestamp': ts}
409
{'error': 'Conflict. Updating.', 'timestamp': ts}
TBW