Skip to content

Latest commit

 

History

History
101 lines (67 loc) · 4.23 KB

README.rst

File metadata and controls

101 lines (67 loc) · 4.23 KB
Raw

Pipeline Examples

Introduction

We provide some example scripts of running FATE jobs with FATE-Pipeline.

Please refer to the document linked above for details on FATE-Pipeline and FATE-Flow CLI v2. DSL version of provided Pipeline examples can be found here.

Quick Start

Here is a general guide to quick start a FATE job.

  1. (optional) create virtual env

    python -m venv venv
source venv/bin/activate
pip install -U pip

  2. install fate_client

    # this step installs FATE-Pipeline, FATE-Flow CLI v2, and FATE-Flow SDK
pip install fate_client
pipeline config --help

  3. configure server information

    # provide ip and port information
# optionally, set logs directory
pipeline config --ip 127.0.0.1 --port 9380

  4. upload data with FATE-Pipeline

    #  upload data used by demo, optionally provides directory where examples/data locates

python demo/pipeline-upload.py --base /data/projects/fate

    If upload job is invoked correctly, job id will be printed to terminal and a upload bar is shown. If FATE-Board is available, job progress can be monitored on Board as well.

    UPLOADING:||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||100.00%
2020-10-16 10:44:26.578 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016104426367594590
Job is still waiting, time elapse: 0:00:01
Running component upload_0, time elapse: 0:00:03
2020-10-16 10:44:31.042 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016104426367594590
2020-10-16 10:44:31.043 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:04

  1. run FATE-Pipeline fit and predict jobs

    python demo/pipeline-mini-demo.py

    As job runs, progress of job execution will also be printed as modules are executed. A message indicating final status ("complete") will also be printed at the end of the job.

    2020-10-16 13:14:56.316 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016131456016425640
Job is still waiting, time elapse: 0:00:00
Running component reader_0, time elapse: 0:00:03
Running component dataio_0, time elapse: 0:00:05
Running component intersection_0, time elapse: 0:00:10
Running component hetero_lr_0, time elapse: 0:00:36
2020-10-16 13:15:33.703 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016131456016425640
2020-10-16 13:15:33.703 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:37

    Once fit job completes, demo script will print coefficients and other training information of model.

    After having completed the fit job, script will invoke a predict job with the trained model. Note that Evaluation component is added to the predict job workflow. For more information on using FATE-Pipeline, please refer to this guide.

    2020-10-16 13:15:34.282 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:125 - Job id is 20201016131533727391641
Job is still waiting, time elapse: 0:00:02
Running component reader_1, time elapse: 0:00:05
Running component dataio_0, time elapse: 0:00:07
Running component intersection_0, time elapse: 0:00:12
Running component hetero_lr_0, time elapse: 0:00:17
Running component evaluation_0, time elapse: 0:00:23
2020-10-16 13:15:58.206 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:133 - Job is complete!!! Job id is 20201016131533727391641
2020-10-16 13:15:58.207 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:230-10-16 10:46:09.350 | INFO     | pipeline.utils.invoker.job_submitter:monitor_job_status:134 - Total time: 0:00:23