Skip to content

Releases: mozilla-ai/lumigator

v0.1.2-alpha

07 Mar 10:41
@agpituk agpituk
v0.1.2-alpha
794887e
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.2-alpha Latest
Latest

Lumigator v0.1.2-alpha Release Notes

Features 🚀

  • UI – Dataset preview: View dataset contents in a tabular format. The dataset viewer supports searching/filtering and enables exporting the data to CSV for easy analysis.
  • Secret Management API: Added API endpoints to list and delete stored Secrets. You can now retrieve a list of configured secret names (with descriptions) and remove secrets via the Lumigator API.
  • New task, Translation in API and SDK: You can now evaluate causal models for translation using datasets with ground truth. Feature available through API and SDK (UI coming soon).
  • New LLM as a judge metric: GEval.
  • Support of any OpenAI-compatible API with LiteLLM.

Bug Fixes 🐞

  • MLflow Tracking URI Requirement: Fixed an issue where a missing MLflow tracking URI would cause a runtime error. Lumigator now properly checks for MLFLOW_TRACKING_URI and throws a clear error if it’s not set.
  • Database Path on Local Deployments: Corrected the default SQLite database file location for local installs. Previously, the local database was not being created in the intended path. The default DATABASE_URL has been fixed to ensure the SQLite file is stored in the db-data/ volume as expected.
  • UI Consistency: Resolved a minor UI issue where the “View Results” button for experiments could appear disabled even after a successful run. Now, experiment result links are enabled as expected once a workflow succeeds (allowing you to inspect outputs immediately).
  • Other stability fixes and tweaks across the stack (including error message corrections and test fixes) improve Lumigator’s reliability. For example, error handling in the backend was refined to bubble up meaningful messages, and several unit tests were fixed to reflect the latest interface changes.

Refactoring 🔧

  • Sample Data Organization: Restructured the built-in sample datasets and related code. Sample CSV files are now organized by task category (e.g. under a summarization/ subfolder), which makes the project structure more logical. Corresponding test fixtures and documentation references were updated to match the new file locations.
  • Unified Task Definition: Refactored how task information is represented in experiment workflows. The code now uses a unified TaskDefinition model (instead of separate “task” and “task type” fields) for experiments and workflows.
  • Code Cleanup: Various parts of the codebase were cleaned and restructured for clarity. For example, the front-end code was streamlined by introducing a generic table component used for both dataset viewing and experiment/job listings, reducing duplicate code. Unused legacy code (like the old “completions” routes) has been removed, and function/class naming has been improved for consistency.

Documentation Updates 📝

  • User Guide: Preparing your own dataset for summarization evaluation
  • README includes details on the optional model caching mechanism and how to configure it, so users can take advantage of faster startup times on first run. Additionally, notes were added on managing secrets with the new API endpoints (so you can see how to list or delete secrets via HTTP requests).
  • API Reference: Regenerated the OpenAPI/Swagger documentation to include the latest API additions and changes.
  • Operational Guides: Minor improvements, e.g. instructions for shutting down Lumigator and troubleshooting sections were expanded with the new error message for a missing MLflow URI.

Infrastructure & CI ⚙️

  • Model Caching for Faster Startup: Implemented a mechanism to pre-download common model files into the local cache on startup. This significantly reduces the “cold start” time for the first experiment run. In practice, when you launch Lumigator or run the test suite, popular Hugging Face models are fetched in advance (controlled by a config flag ENABLE_FIRST_TIME_CACHE), so your initial evaluations kick off more quickly without waiting for large model downloads mid-experiment.
  • Improved Local Dev Workflow: The Makefile targets for managing the local stack were improved. The stop-lumigator and local-down commands now correctly use the Docker Compose local profile to ensure all services (backend, Ray, Minio, etc.) shut down cleanly. This prevents orphaned containers or network issues when you restart Lumigator, especially when switching between configurations.
  • Continuous Integration: Added new integration tests for translation workflows and adjusted CI configurations accordingly. We pinned certain dev dependencies (like IPython) to stable versions to fix test failures in CI. The CI pipeline was also optimized – for instance, steps to run integration tests on SQLite and PostgreSQL now run on a dedicated runner to speed up feedback. These changes make the test suite more robust and ensure that contributions continue to pass checks consistently.
  • Container & Deployment: Updated the Docker compose files and Helm charts to incorporate the latest changes. The Helm chart’s default values were bumped to use v0.1.2-alpha images, and minor tweaks (like adjusting the Ray cluster environment variable name) from this cycle were integrated. Multi-architecture support for the Docker build remains in place and was verified on this release (ensuring Lumigator’s Docker images work on both x86_64 and ARM64).

New Contributors

Full Changelog: v0.1.1-alpha...v0.1.2-alpha

Release Notes created partially with Perplexity.ai.

Contributors

lucacorbucci
Assets 2
Loading

v0.1.1-alpha

18 Feb 16:28
@agpituk agpituk
v0.1.1-alpha
b9f3b84
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.1-alpha

Summary

UI:

  • Introduced TypeScript & general fixes.
  • Prepared the project for the introduction of E2E testing.
  • Added support for the new workflows endpoint.
  • Improved error messages
  • Added sample dataset link for people to quickly find and use datasets in experiments

SDK:

  • Updated SDK to use the new workflows endpoint

API:

  • Added support for local models via llamafile, ollama or vLLM.
  • Added support for DeepSeek API.
  • Added support for Hugging Face text-generation pipelines.
  • Decouple service (jobs & experiments) layer from the presentation layer (extracted HTTP exceptions).
  • Delete the redundant /completions route and consolidate LLM inference logic into the inference job.
  • Added job_result_download endpoint to the experiment service, allowing direct retrieval of job outputs.

Infrastructure:

  • Added persistence to local environments (database & ray).
  • Fixed multiplatform builds
  • Fixed issues with ray dead nodes in local environment after restarting the project.
  • Removed kuberay from the helm installation.

Documentation:

  • Improved helm documentation.
  • Added a guide for running inference with Lumigator on any locally hosted model (via ollama, vLLM and llamafile).
  • Improved documentation on how to run unit and integration tests.
  • Added general documentation on environment variables.

General changes:

  • Reorganized paths and folders within the project.
  • Added MLFlow to track experiments.
  • Reworked the environment variables flow adding a new build process.

PRs

New Contributors

Full Changelog: v0.1.0-alpha...v0.1.1-alpha

Contributors

peteski22, chainlink, and 11 other contributors
Loading

v0.1.0-alpha

07 Feb 17:04
@ividal ividal
v0.1.0-alpha
f80d386
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Verified
Learn about vigilant mode.
Compare
Choose a tag to compare
v0.1.0-alpha

What's Changed

Roadmap features

[FEATURE] User can install Lumigator in their k8s cluster
[FEATURE] User can select multiple models to compare on the same dataset
[FEATURE] User can launch an experiment for summarization with a dataset that did not have ground-truth
[FEATURE] Evaluate experiment results
[FEATURE] User can launch an experiment for summarization that already has ground-truth
[FEATURE] Save inference output as a dataset which can be downloaded

Summary

New UI:

  • Added detailed display for datasets and experiments.
  • Enabled dataset download and results viewing.
  • Improved job status updates and error handling.
  • Added support for multiple models in experiments.
  • Enabled ground truth generation (annotations) and improved dataset validation.

New SDK:

  • Added support for experiments, datasets and launching annotation jobs.
  • Introduced health endpoints and transparent error handling.
  • Vendor caching adjustments.
  • Packaged SDK plus integration tests.

API changes:

  • Refactored jobs endpoints (/health, /jobs, /experiments).
  • Exposed available models via endpoint (/models).
  • Added support for CORS API requests and job status refreshes.
  • Introduced alembic for database migrations and improved error handling.
  • Added support for inference with seq2seq models (via Transformers) and opinionated annotations for summarization with BART-large-cnn.

Infrastructure:

  • Migrated from Pants to UV for building and testing.
  • Added Docker support for local development and deployment.
  • Introduced Helm charts for backend and frontend services.
  • Improved CI/CD pipelines for building and publishing Docker images and PyPI packages.
  • Added support for Nvidia GPUs and debugging tools like Debugpy.

Documentation:

  • User documentation and guides: https://mozilla-ai.github.io/lumigator/
  • Improved documentation for dataset upload and model data.
  • Added troubleshooting sections.
  • Updated contributing guidelines and issue templates.

PRs

Read more

Contributors

peteski22, chainlink, and 15 other contributors
Loading
0 Join discussion