This repository contains a RESTful API to interact with a cloud target platform for federated machine learning. It is primarily targeted at algorithm developers who are planning to design and run federated machine learning algorithms through a graphical user interface.
The graphical user interface (front-end component) it is available at "https://github.com/Engineering-Research-and-Development/musketeer-client-connector-frontend".
In order to run the application through a Docker container, a Docker installation is required in the host. The Docker Daemon must be running to perform the following commands.
To enable the direct download of the Docker images created for the Muskeeter's end users, you should log in to our registry using the following commands:
-
docker login gitlab.alidalab.it:5000/musketeer/connector-client-backenduser and password: contact musketeer-team@eng.it -
docker login gitlab.alidalab.it:5000/musketeer/ngx-musketeer-clientuser and password: contact musketeer-team@eng.it
As a first step create the Docker image of the backend components. From the project root, run the following command through the terminal:
docker build -f Dockerfile -t MYBUILDIMAGE
The same MYBUILDIMAGE name chosen must be inserted in the docker-compose.yml file.
Before running the docker-compose.yml the user must also configure the Docker volumes for the backend component. In particular, it is necessary to specify:
YOUR_LOCAL_DATA_PATH: a filesystem path directory where there are the datasets that you want to bind to the Client Connector.
YOUR_LOCAL_RESULTS_PATH: a filesystem path directory where to store all the results file generated by the task you run and complete.
The docker-compose.yml contains both the backend image just created and the frontend component. The frontend Docker image is located on a repository accessible through authentication to our Docker registry.
The docker-compose.yml contains also a MongoDB image to store mainly the local metadata information provided by the user during the dataset registration. It is no needed to modified this service image specification.
To log in, run the following command: docker login gitlab.alidalab.it:5000/musketeer/ngx-musketeer-client, followed by USER and PASSWORD that have been provided by ENGINEERING TEAM.
Finally, run the following command to run and up the Desktop Client Connector:
docker-compose pull && docker-compose up
The frontend Docker image will be automatically pulled from the register, if it is not present. It may take some minutes to download all the required dependencies based on your internet connection. Once it is done, the local server will be running at '127.0.0.1:5000', whilst you can use the User Interface by opening a browser and writing the following URL: '127.0.0.1:4500' (or 'localhost:4500').
If this is the first time you start the Client Connector you will have to configure both the server communication and Federated Machine Learning libraries part. How to configure the Client Connector to be used within the project MUSKETEER by the pilots is shown below.
Git url: "git+https://github.com/IBM/pycloudmessenger.git@master" Module: pycloudmessenger.ffl.fflapi Configurations file: config.json file (the configuration file provided you by IBM to access the Musketeer core server)
Git url: "git+https://github.com/Musketeer-H2020/MMLL.git" (please set also your GitHub personal access token (PAT) to download the MML library) Aggregator Classpath: MMLL.nodes.MasterNode.MasterNode Participant Classpath: MMLL.nodes.WorkerNode.WorkerNode Aggregator Wrapper Classpath: MMLL.comms.comms_pycloudmessenger.Comms_master Participant Wrapper Classpath: MMLL.comms.comms_pycloudmessenger.Comms_worker Catalogue file: catalogue.json file (it contains the definition of the algorithms provided and their properties)
The repository contains the "http cloud messenger" library that implements the communication interface used within the MUSKETEER project, and which can be used to communicate securely within an IDS (International Data Space) ecosystem. The library in question, in fact, can communicate with another component via HTTP protocol; in particular, it can communicate with another optional component within the Client Connector, the "musketeer-client-connector-data-app", available on "LINK" to generate and consume data on top of an "Execution Core Container" (ECC). The library can be installed via the configuration steps of the Client Connector and be used in addition to the new components that are part of the so-called "true-connector" (https://github.com/Engineering-Research-and-Development/true-connector) implemented by ENGINEERING for the communication of nodes within an IDS ecosystem. The Client Connector will thus be able to consume and produce data with a server node or other edge nodes within an IDS ecosystem.
To use the Client Connector within an IDS ecosystem, use the "docker-compose.yml" inside the "ids" directory containing, in addition to the back-end, front-end and MongoDB Docker micro-service, the following components:
1) Execution Core Container (ECC), open-source project designed by ENG. It is in charge of the data exchange through the IDS ecosystem representing data using the IDS Information Model and interacting with an external Identity Provider. It is also able to communicate with an IDS Broker for registering and querying information. (https://github.com/Engineering-Research-and-Development/true-connector-execution_core_container)
2) Musketeer-Back-End (MBE) Data Application; it represents a data application for generating and consuming data on top of the ECC component towards the MUSKETEER scope. (https://github.com/musketeer-eng-team/true-connector-basic_data_app)
3) Usage-Control (UC) Data Application, a customised version of the Fraunhofer IESE base application for integrating the MyData Framework (a Usage Control Framework designed and implemented by Fraunhofer IESE) in a connector. (https://github.com/Engineering-Research-and-Development/true-connector-uc_data_app)
The back-end Client Connector component uses the HTTP cloud messenger library mentioned above to communicate with the new components for the IDS ecosystem; as a configuration file, it contains the pointers to the data-app client connector and the producer connector, in this case as an internal testing. In fact, for testing purposes, this docker-compose file contains three other components that simulate the behaviour of a Server-Connector ("producer connector"). In this way it is possible to test the following invocations (for example by using "curl" commands) which start from the Client-Connector-backend up, pass through the new components to make them a communication for IDS, arrive at the "producer" component which will respond with an appropriate testing message:
1) curl --location --request POST 'http://localhost:5000/cc/comms/login' \
--header 'Content-Type: application/json' \
--data-raw '{"user": "test_user",
"password": "test_password"}'
2) curl --location --request GET 'http://localhost:5000/cc/comms/tasks'
You can run the above "curl" commands after you have correctly run the whole Client Connector package running the "docker-compose up" command inside the "ids" directory.
The extra Docker components to simulate the server side, and you can then remove from the docker-compose.yml file, are the following: ecc-provider, uc-dataapp-provider, be-dataapp-provider.
To download the Docker image of the MBE Data Application from the registry 'gitlab.alidalab.it:5000/musketeer/connector-client-backend/httpcm/be:1.0.1' please contact us at 'musketeer-team@eng.it'.
CATALOGUE
Get the list of algorithms provided during the Client Connector configuration steps.
GET /cc/catalogue/algorithms
Get the list of privacy operation modes (POMs) provided towards the MUSKETEER project.
GET /cc/catalogue/poms
CLIENT CONNECTOR / CONFIGURATION
Get a number indicating the configuration status of the Client Connector: 1 - to install the communication and FML algorithms library; 2 - to install the FML algorithm library; -1 - Client Connector fully configured with the libraries installed.
GET /cc/configurations/step
Add the communication library configuration to integrate into the Client Connector. Required information: - comms_git_url: Git URL where to download the comms package - comms_git_token: Git Token to access private repository - comms_module: module name to import - comms_config: JSON configuration for the comms instance used
POST /cc/configurations/comm
{
comms_git_url: str,
comms_git_token: str
comms_module: str,
comms_config: dict
}
Add the federated machine learning library configuration to integrate into the Client Connector. Required information: - mmll_git_url: Git URL where to download the Federated Machine Learning package - mmll_masternode_classpath - mmll_workernode_classpath - mmll_comms_wrapper_classpath - mmll_algorithms: JSON file containing the algorithms specifications and details
POST /cc/configurations/mmll
{
mmll_git_url: str,
mmll_masternode_classpath: str,
mmll_workernode_classpath: str,
mmll_comms_wrapper_classpath: str,
mmll_algorithms: dict
}
Get the communication library configuration
GET /cc/configurations/comm
Get the Federated Machine Learning library configuration
GET /cc/configurations/mmll
Get a chart image resulting from the execution and completion of a Federated Machine Learning task
GET /cc/results/image?task
Get the log event stream related to a specified task you are participating/aggregating to
GET /cc/results/stream/logs?task&mode
Get your list of dataset metadata you have registered through the Client Connector
GET /cc/datasets
Register a new dataset into the Client Connector specifying the required information, depending on the type of storage used (e.g. local or database), and the data format (e.g. CSV or PKL)
POST /cc/datasets
{
type: str,
spec:
{
"name": str,
"path": str,
"format": str,
...
}
}
Delete a specified dataset metadata through its unique identifier assigned to it
DELETE /cc/datasets/<_id>
Modify/Update a specified dataset metadata through its unique identifier assigned to it, and inserting the updated information to modify.
PUT /cc/datasets/<_id>
{
type: str,
spec:
{
"name": str,
"path": str,
"format": str,
...
}
}
COMMUNICATIONS
Access to the target platform using the configured communication messenger library
POST /cc/comms/login
{
user: str,
password: str
}
Logout from the target platform
POST /cc/comms/logout
Register an account to the target platform using the configured communication messenger library
POST /cc/comms/registration
{
user: str,
password: str,
org: str
}
Change the password access credential to the target platform using the configured communication messenger library
PATCH /cc/comms/change_password
{
new_password: str
}
Deregister the logged-in user from the target platform using the configured communication messenger library
DELETE /cc/comms/deregister
Get the list of the created tasks in the target platform using the configured communication messenger library
GET /cc/comms/tasks
Create a new task by inserting the name of the task and all required fields that define a task towards the configured communication messenger and federated machine learning libraries
POST /cc/comms/tasks
{
task_name: str,
definition: dict
}
Get the task information of a specified task using the configured communication messenger library; in addition to the task specification, another information, useful for the GUI, is included so that action icons are shown or not depending on the logged-in user. For instance, only the task creator can aggregate or delete his own task.
GET /cc/comms/tasks/<task_name>
Delete a specified task in the target platform using the configured communication messenger library
DELETE /cc/comms/tasks/<task_name>
Get the list of the tasks created by the user in the target platform using the configured communication messenger library
GET /cc/comms/tasks/created
Get the list with all the joined tasks in the target platform using the configured communication messenger library
GET /cc/comms/tasks/joined
Get the list of all the tasks the user is participating in the target platform using the configured communication messenger library
GET /cc/comms/tasks/assigned
Get the list with all the available trained models in the target platform using the configured communication messenger library
GET /cc/comms/models
Requests a trained model, related to a specified task, in the target platform using the configured communication messenger library; the object obtained is then downloaded and saved locally by selecting one of the three extensions (formats) chosen: PKL, PMML, ONNX. An error message will be sent if a particular extension is not supported by the model
GET /cc/comms/models/<task_name>?extension
Requests a model deletion from the selected task in the target platform using the configured communication messenger library
DELETE /cc/comms/models/<task_name>
Requests the model lineage, related to a specified task, in the target platform using the configured communication messenger library
GET /cc/comms/models/<task_name>/lineage
FEDERATED MACHINE LEARNING
As a task owner, aggregate to a task by entering the information needed to access a validation and test dataset. Then run the script that will execute the Federated Machine Learning algorithm as aggregator defined in the task towards the configured and installed Federated Machine Learning library.
POST /cc/fml/aggregate
{
task_name: str,
datasets: dict
}
data = request.json
As a participant, join a task by entering at least the information needed to access a training dataset, and optionally a validation and test dataset. Then run the script that will execute the Federated Machine Learning algorithm as participant defined in the task towards the configured and installed Federated Machine Learning library.
POST /cc/fml/participate
{
task_name: str,
datasets: dict
}
This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 824988. https://musketeer.eu/