The laboratory service is responsible for managing all the architecture and laboratory files generated by Flogo users, as well as the datasets they intend to use. Its main objective is to facilitate the management of all the model training tests that users wish to perform with the frameworks that comprise Flogo. To this end, it provides users with their own repository where they can store all generated files. Additionally, this service allows the execution of the uploaded laboratories through a simple API call, managing all the results generated during the training of an architecture, including models and their statistics.
This service can be very useful as it alleviates many responsibilities from the user. On one hand, it automates the management of all generated files, eliminating the need to do it manually. Additionally, it prevents developers from having to manually access the training results and the obtained models, as the server provides them through API calls.
In this repository, you will find all the Java classes developed to support the described service. Additionally, you will find the necessary steps to run this service locally and a user manual for it.
The steps to follow to use the service are as follows:
-
The first step is to download the file service.zip. This can be done by cloning the entire repository or simply downloading the specified zip file.
-
Next, you should unzip the file in the directory where you want to run the service. Note that this will create directories and files, so it should not be placed in a location protected by permissions. This process may take a few minutes.
-
The unzipped directory contains three files: two jar files corresponding to the service and the interface that allows easy communication with it, and the directory that contains everything necessary, except the Python enviroment, for the correct execution of the program.
-
The next step is to create the necessary Python virtual environment to run the laboratories provided. To do this, execute the following instruction while in the Flogo directory:
python -m venv execute/flogo_env
- Once the environment is created, you need to install the required libraries. Execute the following command in the same directory:
execute/flogo/bin/pip install -r execute/requirements.txt
- Posteriormente, incluimos las clase de Flogo dentro del entorno virtual. Se tiene que ejecutar las siguientes instrucciones.
cp ./execute/flogo_classes/implementations execute/flogo/Lib/site-packages/implementations
cp ./execute/flogo_classes/framework execute/flogo/Lib/site-packages/framework
- Subsequently, include the Flogo classes within the virtual environment. To do this, you need to execute the following instructions:
java -jar service.jar /ruta/al/directorio/flogo
Once the service is launched, it can be used through API calls. To facilitate the work for end users, a program was developed that acts as an interface, considerably simplifying these procedures. Each API call will be made by executing the file api-interface.jar. The possible operations that can be performed are as follows.
- File upload: This method will be used by the user whenever they want to upload an object, whether it be an architecture, a laboratory, or a dataset, as it simplifies the process of uploading files to the repository through the API. This is particularly useful because adding a file, regardless of its extension, to the body of a request can be a complex procedure. The interface handles this process, relieving the user of this responsibility. To perform this operation, three parameters must be provided to the interface. The first is the word "post," which is reserved for this operation. The second parameter specifies the type of object to be uploaded, which can be an architecture, a laboratory, or a dataset. The third parameter indicates the path where the file to be uploaded is located. It should be noted that, in the case of datasets, the directory containing all the files necessary for its declaration in the operational framework must be provided.
java -jar api-interface.jar post architecture /ruta/fichero/architecture.py
- Getting file: This function will be responsible for requesting and downloading the file that the user has requested, storing it in the location provided as a parameter. It will be used when the developer wants to retrieve any of the files uploaded to the service, an essential operation in any repository. To execute this operation correctly, the client must supply several parameters. The first must be "get," which is the word reserved for this operation. Next, they must specify the type of object they want to download and its name through the second and third parameters, respectively. The last parameter will indicate the path where the downloaded object will be saved. In case a dataset is to be downloaded, it is important that the file where it will be saved has a ZIP extension.
java -jar api-interface.jar get architecture architecture_name /ruta/fichero/architecture.py
- List files: Obtaining a list of files that have been uploaded to the repository is crucial for knowing the status of the file storage at any given moment. To perform this task, we have this method in the interface. Its response will show the name of all objects of the specified type that are in the repository. For the execution of this method, only two parameters are necessary. The first one, as in all cases, is used to indicate the reserved word for the operation you want to perform. In this case, the word is "list." The second parameter will be used to indicate the type of object for which you want to obtain the list.
java -jar api-interface.jar get list architecture
- File deletion: This function performs the last of the main operations within a repository, as it allows deleting files that have been uploaded. This is essential for users to have complete control over all their files within the storage. For its correct execution, only three simple parameters are necessary. The reserved word for this method is "delete," followed by the type of object to be deleted and its name, indicated in this order.
java -jar api-interface.jar delete architecture architecture_name
- Lab execution: When training any of the uploaded architectures, this method will be invoked. For this purpose, the specified laboratory will be executed. This function is the most important of all supported by the interface, as it allows for the training of a neural network in a very straightforward manner. The execution call has only two parameters. The first one is common to all functions, used to indicate which operation is desired through a reserved word. In this case, it is "execute". Subsequently, it will only be necessary to indicate the laboratory to be executed, and the training process will begin.
java -jar api-interface.jar execute laboratory_name
- Getting stat: This method allows users to obtain various statistics or stats provided by the service, showing how the laboratory executions have progressed. These statistics are crucial as they offer relevant information about the experiments tested and, therefore, about the sets of hyperparameters used. To invoke each of the stats, at least two parameters must be provided. These are the reserved word for this operation, which is "stat," and the statistic to be obtained from those available.
java -jar api-interface.jar stat stat_name [laboratory=laboratory_name] [experiment = experiment_name]
- Getting model: Users should have the ability to obtain models that have been trained. Therefore, this function will provide a ZIP file to users containing all the necessary files to run a Docker image with the specified model. With this, developers can incorporate their trained neural networks into their applications or use them as they see fit. Four parameters are required: the first one will be "model," which is the reserved word for this operation. The second and third parameters will be the laboratory and the experiment from which the model is desired, respectively.
java -jar api-interface.jar model architecture_name /ruta/fichero/model.zip
This project is licensed under the EUPL v1.2 License - see the LICENSE file for details.
(c) 2024 José Juan Hernández Gálvez
Github: https://github.com/josejuanhernandezgalvez
(c) 2024 Juan Carlos Santana Santana
Github: https://github.com/JuanCarss
(c) 2024 Joel del Rosario Pérez
Github: https://github.com/Joeel71