Python programmatic interface

pfconclient Python3 programmatic interface

The pfconclient package provides a client module with eight main methods among others:

1. run_job: Run plugin until finished and get the resulting files in a local directory
2. submit_job: Submit plugin for execution and return without waiting for the plugin to finish
3. poll_job_status: Keep polling for the execution status of a previously submitted plugin until it finishes
4. get_job_status: Perform a single check on the execution status of a previously submitted plugin
5. get_job_zip_data: Download the output files of a previously submitted plugin that has already finished as a single zip file and return the content of the zip file
6. get_job_zip_file: Download the output files of a previously submitted plugin that has already finished as a single zip file and save the file into a local directory
7. get_job_files: Download the output files of a previously submitted plugin that has already finished as a single zip file and unpack the zip file's content files within a local directory
8. delete_job: Delete a previously submitted plugin

Instantiate the client:

from pfconclient import client

token = client.Client.get_auth_token('http://localhost:30006/api/v1/auth-token/', 'pfcon', 'pfcon1234')
cl = client.Client('http://localhost:30006/api/v1/', token)

Methods usage

run_job

Run fs plugin until finished using any local input directory and get the resulting files in a local output directory:

job_descriptors = {
    'cmd_args': '--saveinputmeta --saveoutputmeta --dir cube/uploads',
    'cmd_path_flags': '--dir',   # flags with arguments of type 'path' or 'unextpath' should be specified here separated by comma
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simplefsapp',
    'selfexec': 'simplefsapp',
    'selfpath': '/usr/local/bin',
    'execshell': 'python3',
    'type': 'fs'
    }
job_id = 'chris-jid-20'
inputdir = '/tmp/sbin/in'
outputdir = '/tmp/sbin/out/chris-jid-20'
cl.run_job(job_id, job_descriptors, inputdir, outputdir)

Run ds plugin until finished using the local output directory of a previous plugin as its input directory and get the resulting files in a local output directory:

job_descriptors = {
    'cmd_args': '--saveinputmeta --saveoutputmeta --prefix lolo',
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simpledsapp',
    'selfexec': 'simpledsapp',
    'selfpath': '/usr/local/bin',
    'execshell': 'python3',
    'type': 'ds'
    }
job_id = 'chris-jid-21'
inputdir = '/tmp/sbin/out/chris-jid-20'
outputdir = '/tmp/sbin/out/chris-jid-21'
cl.run_job(job_id, job_descriptors, inputdir, outputdir)

submit_job

Submit fs plugin for execution using any local input directory and return without waiting for the plugin to finish:

job_descriptors = {
    'cmd_args': '--saveinputmeta --saveoutputmeta --dir cube/uploads',
    'cmd_path_flags': '--dir',  # flags with arguments of type 'path' or 'unextpath' should be specified here separated by comma
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simplefsapp',
    'selfexec': 'simplefsapp',
    'selfpath': '/usr/local/bin',
    'execshell': 'python3',
    'type': 'fs'
    }
job_id = 'chris-jid-22'
inputdir = '/tmp/sbin/in'
cl.submit_job(job_id, job_descriptors, inputdir)

Submit ds plugin for execution using the local output directory of a previously finished plugin as its input directory and return without waiting for the plugin to finish:

job_descriptors = {
    'cmd_args': '--saveinputmeta --saveoutputmeta --prefix lolo',
    'auid': 'cube',
    'number_of_workers': 1,
    'cpu_limit': 1000,
    'memory_limit': 200,
    'gpu_limit': 0,
    'image': 'fnndsc/pl-simpledsapp',
    'selfexec': 'simpledsapp',
    'selfpath': '/usr/local/bin',
    'execshell': 'python3',
    'type': 'ds'
    }
job_id = 'chris-jid-23'
inputdir = '/tmp/sbin/out/chris-jid-20'
cl.submit_job(job_id, job_descriptors, inputdir)

poll_job_status

Keep polling for the execution status of a previously submitted plugin until it finishes:

job_id = 'chris-jid-22'
l_status = cl.poll_job_status(job_id)

get_job_status

Perform a single check on the execution status of a previously submitted plugin:

job_id = 'chris-jid-23'
l_status = cl.get_job_status(job_id)

get_job_zip_data

Get a job's zip file byte content:

job_id = 'chris-jid-22'
zip_content = cl.get_job_zip_data(job_id)

get_job_zip_file

Get and save a job's zip file into a local directory:

job_id = 'chris-jid-23'
local_dir = '/tmp/sbin/out/chris-jid-23'
cl.get_job_zip_file(job_id)

get_job_files

Get a job's output files unpacked within a local directory:

job_id = 'chris-jid-23'
local_dir = '/tmp/sbin/out/chris-jid-23'
cl.get_job_files(job_id)

delete_job

Delete a previously submitted plugin:

job_id = 'chris-jid-23'
cl.delete_job(job_id)