NOTE:
mainbranch now tracksv2by default. To access the original Bailo, see thev1branch.v1is in the process of being removed from this project, see migration for more information.
Making it easy to compliantly manage the machine learning lifecycle
Explore the docs »
Report a Bug
·
Request a Feature
Table of Contents
Bailo helps you manage the lifecycle of machine learning to support scalability, impact, collaboration, compliance and sharing.
- Node v18
- Docker / Docker Compose
To run in development mode (modified files on your host machine will be reloaded into the running application):
git clone https://github.com/gchq/Bailo.git && cd Bailo
npm install
npm run certs
# This builds all the Bailo images, rerun it when you update dependencies.
docker compose build --parallel
# Then run the development instance of Bailo.
docker compose up -dOn first run, it may take a while (up to 30 seconds) to start up. It needs to build several hundred TypeScript modules. These are cached however, so future starts only require a few seconds. You should access the site via localhost:8080.
The registry requires a JWKS file for the token authentication with the backend application. For development, a JWKS
file is generated by running npm run certs. For production, the script generateJWKS.ts can be used to generate a
JWKS file for the public key referenced in the backend application configuration.
Some example schemas are installed by default. More schemas can be added by altering and running the
addDeploymentSchema.ts and addUploadSchema.ts files.
npm run script -- addDeploymentSchema
npm run script -- addUploadSchemaNOTE: Scripts are also written in Typescript. In production, run them using
node, in development, run them usingts-nodeornpm run script.
| Service | Host | Notes |
|---|---|---|
| Next UI | 3000 | Stored in frontend |
| NodeJS App | 3001 | Stored in backend |
| Mongo | 27017 | mongoadmin:mongoadmin |
| Registry | 5000 | HTTPS only, no UI |
| Minio UI | 9001 | minioadmin:minioadmin |
| Minio | 9000 | minioadmin:minioadmin |
| MailCrab | 1080 | Fake email server |
** Note: these credentials are intentionally basic/default, but in your own instances we recommend changing them to something more secure.
We expect the administrator to provide their own forms of authentication. By default all users authenticate using as 'user'.
You can test out your new deployment using the example models which can be found in __tests__
minimal_binary.zip and
minimal_code.zip. There are also example forms in the scripts folder
minimal_upload_schema_examples.json and
minimal_deployment_schema_examples.json.
- A user accesses a URL. We use NextJS routing to point it to a file in
frontend/pages.[xxx].tsxfiles accept any route,xxx.tsxfiles allow only that specific route. - Data is loaded using SWR. Data loaders are stored in
./frontend/data. Each one exposes variables to specify if it is loading, errored, data, etc. - Requests to the backend get routed through express within
backend/routes.ts. Each route is an array with all items being middleware except the last, which is the handler ([...middleware, handler]). - Routes interact with the database via
mongoose, which stores models in./backend/models.
Some processing is done away from the main thread, when it is expected to take longer than a few milliseconds. These are
posted to a mongodb queue and processed by handlers in the backend/processors folder. Mongodb queues are handled
invisibly by p-mongo-queue (backend/utils/queues.ts).
Issue: Sometimes Docker struggles when you add a new dependency.
Fix: Run docker compose down --rmi all followed by docker compose up --build.
Issue: Sometimes SWR fails to install its own binary and the project will refuse to start up (development only)
Fix: Run npm uninstall next && npm install next. Some users report still having issues. If so, run:
rm -rf node_modules && rm -rf package-lock.json && npm cache clean -f && npm i.
Issue: Unable to authenticate to the Docker registry / compile binaries.
Fix: Make sure that your authentication proxy is setup to allow the 'Authorisation' header. Make sure that your application is able to access the Docker registry internally as it will not provide user authentication.
List of near term goals:
- Model metrication
- Instance federation
28-04-2022: Changed 'schema' model to be stored as a string instead of an object. Delete & recreate your schemas.
Bailo is released under the Apache 2.0 Licence and is covered by Crown Copyright. See LICENSE.txt for more
information.