Overview

Propius is a Federated Learning (FL) resource manager. You can define a FL job, specifying number of clients (edge devices) the job requires, and number of estimated job round. Propius can then route all the clients you need back to you! Additionally, you can set constraints about the device specification (OS version, memory size, etc) for your job so that your FL task can run smoothly on the edge devices, and Propius can make sure that your requirement will be satisfied!

Propius uses GRPC protocal for communication, and Redis for data storage.

Scheduler

Scheduler is the componenent that gives a priority score to every FL job, so that multiple jobs active in Propius can be scheduled with edge device resources. Currently, Propius supports FIFO, Random, SRJF, and IRS, our resource-aware algorithm that has the goal to speed up your FL job!

Job manager

Job manager handles incoming job request. It parses the requirement sent from the job provider, and stores the processed job information in the job database so that client manager can later fetch the job and assign it to clients. Also, job manager will inform the scheduler that a new job has arrived, and let the scheduler update the score for this job and other jobs.

JOB_REGIST

At the beginning of a FL job lifetime, job_info should be sent to job manager, specifying:

1. the parameter server address (ip, port) that hosts the FL job
2. public_constraint of the job on clients (edge device). Public constraints should be constraints on client device specifications that are not sensitive in nature, and client devices are able to provide those information to Propius. Propius maintains a list of constraint options for job to selectively choose and define.
3. private_constraint of the job on clients (edge device). Private constraints are constraints on sensitive client specifications. Propius maintains a list of constraint options for job to selectively choose and define.
4. est_demand (estimated job round demand) of the FL job. Though the job provider can change demand at the start of each FL round, Propius asks for a estimated round demand of each job to make better scheduling decisions
5. est_total_round (estimated number of rounds in total) of the FL job. Though the job provider can terminate the FL job earlier (before the actual round number equals to this total round number), Propius acks for this estimation to make better scheduling decisions. Please note that this estimation would also serve as the maximum round service Propius is able to provide to this FL job. If the actual round exceeds this number, Propius will terminate its service to ensure fairness.

Job manager will return a job_register_ack message to job, indicating its job IF, and register status boolean.

JOB_REQUEST

Job should make a request to Propius job manager at the start of each round to inform Propius that the job is ready for new clients. Propius will then start to assign this job to eligible clients until the demand for this round is satisfied. Job should provide:

1. id: the ID of the job assigned by Propius earlier during the register stage
2. demand: current round demand number for clients

Job manager will return ack message indicating whether the request is successful.

JOB_END_REQUEST

Job is recommended to make a end request to Propius job manager to terminate job scheduling phase of a round, whether the actual client allocation number hits the pre-requested amount. Job need to provide job_id message indicating the ID of the job. If successful, job manager will return acknowledge message (ack=True) message to job.

JOB_FINISH

Job is advised to make a finish request to Propius job manager at the end of the job. Job will be terminated by this call, even when the round hasn't reached the estimated total round. Job need to provide job_id message indicating the ID of the job.

Job database

Client manager

Client database