699 lines (399 loc) · 18.6 KB

Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

1.0.0 - 2024-10-14

No significant changes.

0.43.0 - 2024-09-12

Added

A new service called ProfilingService in charge of propagating profiling
- For now, only register and propagates profiling steps
- Introduce a new asset type, ASSET_PROFILING_STEP (#407)

Fixed

DisableOutput does not panic if there is not output assets associated with an output (#438)

0.42.0 - 2024-06-13

Changed

Rename ComputePlanDBAL().IsPlanRunning(key) to ComputePlanDBAL().ArePlanTaksRunning(key) (#432)

Fixed

Compute plans are identified as not running if a task failed or is canceled, even if some tasks are still in doing. (#427)
IsPlanRunning checks if a cancel of failure date is set before checking the tasks status. (#432)

0.41.0 - 2024-06-03

No significant changes.

0.40.0 - 2024-03-27

Changed

- BREAKING: remove type from datamanager table (#394)
- [chore] towncrier is now used for changelog management (#395)

0.39.0 - 2024-03-07

Changed

Compute task status DOING is renamed EXECUTING (#371)

0.38.0 - 2024-02-26

Removed

BREAKING: remove all code related to the distributed mode, and mentions in schemas and documentation (#341)
BREAKING: distributed Skaffold profile and mentions in doc (#319)
BREAKING: chaincode-init and chaincode Dockerfiles (#319)
Flag & environment variables to choose between standalone and distributed mode (#347)

Added

Add Image addressable to Function object. By default, Image is set to an empty string adressable. This addressable is updated when the checksum and storageAddress are available (#288)
Enum FailedAssetKind (#277)
BREAKING: Field asset_type of type FailedAssetKind in FailureReport (#277)
BREAKING: Add FunctionStatus (#263)
Add Function status event machine (#263)
BREAKING: Add statuses WAITING_FOR_BUILDER_SLOT and BUILDING on tasks to reflect associated function status (#366)
Add task actions BUILD_STARTED and BUILD_FINISHED to propagate status change from function to compute task (#366)

Changed

Rename Function addressable to Archive (#288)
Renamed compute_task_keyby asset_key in FailureReport (#277)
FailureReport now can be reference a ComputeTask or a Function through asset_key + asset_type (#277)
Logic to determine new compute task status takes in account the status of the function. A new task can now be created with the status FAILEDor CANCELLED (if the function reached the corresponding status) (#365)
BREAKING: Transition to status TODO for a given compute task is done after the function is built(#365)
BREAKING: Rename TODO to WAITING_FOR_EXECUTOR_SLOT and WAITING to WAITING_FOR_PARENT_TASKS(#366)

Fixed

incorrect link in documentation (#307)
Skip compute ask permissions checks when the action is propagated from the function status change (#367)

0.37.0 - 2023-10-18

Added

Source code formatting (#313)

Changed

Bump go version to 1.21 (#316)

0.36.1 - 2023-10-06

Added

three_orgs Skaffold profile for standalone orchestrator (#280)

0.36.0 - 2023-09-07

Changed

Replace deprecated ioutil package with os (#269)

0.35.2 - 2023-07-25

Changed

Minor dependency updates. See commit history for more details.

0.35.1 - 2023-06-27

Changed

Minor dependency updates. See commit history for more details.

0.35.0 - 2023-06-12

No changes on the app

0.34.0 - 2023-05-11

Changed

A Performance in now unique regarding a compute task key, a metric key and a compute task output identifier (#197)

Removed

Metric from Performance (#213)

0.33.0 - 2023-03-31

Changed

BREAKING: rename Algo to Function (#139)

0.32.0 - 2023-01-31

Added

Context in fsm calls (#127)
Contributing, contributors & code of conduct files (#123)

Removed

Test Only field for data samples (#116)

0.31.1 - 2023-01-09

Removed

field parent_task_keys in ComputeTask (#112)
view expanded_compute_tasks (#112)

[0.31.0] - 2022-12-19

Changed

bump app version to 0.31.0

Fixed

end-2-end postgres dump upload (server version)

[0.30.0] - 2022-11-22

Removed

(BREAKING) task category
(BREAKING) task specific data

Changed

TASK_UNKNOWN is a valid category
Allow registration of performances on any task category
Update the TLS certificates (#91)

[0.29.0] - 2022-10-03

Added

allow setting gRPC keepalive enforcement policy

Removed

(BREAKING) delete_intermediary_models property in ComputePlan and NewComputePlan
(BREAKING) ModelCategory and associated Model & NewModel fields
(BREAKING) AlgoCategory and associated Algo and NewAlgo fields

Changed

failure reports: Build errors now have a logs address
(BREAKING): Replaced algo by algo_key in ComputeTask

Fixed

000042_reference_task_outputs use compute_task_key instead of asset_key for assetKey

[0.28.0] - 2022-09-26

WARNING: Some migrations in this version are destructive once applied you will not be able to restore algo categories.

Changed

(BREAKING) Algo.category: do not rely on categories anymore, all algo categories will be returned as UNKNOWN
NewAlgo.category: No category is expected
(BREAKING): Replaced algo by algo_key in ComputeTask

Fixed

Task outputs events come after their referenced asset creation event

Removed

Worker field on NewAggregateTrainTaskData, use NewComputeTask.Worker field instead
DisableModel rpc in Model service, use DisableOutput in ComputeTask service instead
CanDisableModel rpc in Model service.

Added

failure_date field to ComputePlan protocol buffer schema
IsComputePlanRunning gRPC method in ComputePlanService

Removed

(BREAKING) compute plan status

[0.27.0] - 2022-09-19

Added

Worker field on NewComputeTask mandatory for tasks without input data

Deprecated

NewAggregateTrainTaskData.worker: use NewComputeTask.Worker field instead

Removed

NewComputeTask.parent_task_keys which was deprecated since 0.26.0
Restriction on algo-task category matching
Restriction on parent task category

[0.26.1] - 2022-09-12

Changed

Images are built using protoc-gen-go v1.18.1, grpc_health_probe v0.4.12 and migrate v1.28.1

Fixed

Disable output RPC on distributed mode

[0.26.0] - 2022-09-07

Removed

(BREAKING) ModelService.GetComputeTaskInputModels, use ComputeTaskAPI.GetInputAssets instead
Test task rank special case: rank is not inherited from parent task anymore
(BREAKING) QueryModels from model service: it was unused and model category will soon be deprecated

Deprecated

NewComputeTask.parent_task_keys is deprecated since parent tasks are determined from task inputs

[0.25.0] - 2022-08-29

Added

New RPC to disable task outputs

Deprecated

ModelService.GetComputeTaskInputModels, use ComputeTaskAPI.GetInputAssets instead

[0.24.0] - 2022-08-22

Fixed

Properly register compute task outputs in distributed mode

Removed

(BREAKING) Remove RabbitMQ

[0.23.0] - 2022-08-17

Added

New service methods to update algo, compute_plan and data manager name
gRPC method to get task input assets

Changed

Prevent duplicate model registration based on task output definition
Switched to zerolog logging library

[0.22.0] - 2022-08-09

Added

Build images with go 1.19
Add a Transient field to the task inputs
Return an error in distributed mode if a stored event has invalid event or asset kind
Associate asset with task output on registration

Removed

Task counts by status from ComputePlan responses

[0.21.0] - 2022-08-01

Added

Introduce gRPC SubscribeToEvents method in distributed mode

Changed

Validate task inputs

Fixed

In standalone mode, lock the events table when inserting events to prevent missing events in SubscribeToEvents gRPC stream

Removed

Category filter from QueryAlgos rpc
Legacy compute task permission fields

[0.20.0] - 2022-07-25

Added

Introduce gRPC SubscribeToEvents method in standalone mode
Dispatch updated asset event on ComputePlan cancellation

Removed

Automatic transition to DONE when registering models or performances.

Changed

updated grpc healthprobe to 0.4.11 in server image
updated rabbitmq/amqp091-go lib to 1.4.0

Fixed

properly ignore mocks when building image locally

[0.19.1] - 2022-07-13

Fixed

SQL query for organization with null address

[0.19.0] - 2022-07-11

Added

Organization hostname in the organization object
CancelationDate in the compute plan object

Fixed

SQL logging was enabled when METRICS_ENABLED flag was passed instead of documented LOG_SQL_VERBOSE
Prevent disabling model if task has only predict or test children
Don't timeout when canceling a compute plan

[0.18.0] - 2022-07-05

Removed

Metadata set in events

Changed

(BREAKING) Removed the MetricKeys property of test tasks in favor of the generic Algo field

[0.17.0] - 2022-06-20

Added

Enable transition to DONE through ApplyTaskAction

[0.16.0] - 2022-06-14

Changed

(BREAKING) rename node to organization

Fixed

allow a worker to cancel a task it does not own

[0.15.0] - 2022-06-07

Added

Introduce Predict task type
Introduce compute task outputs

Changed

use go test to run e2e tests

[0.14.0] - 2022-05-31

Added

Introduce empty compute plan status

Changed

base docker image from alpine 3.15 to alpine 3.16

Fixed

event asset migration

Changed

only update status on task update.

[0.13.2] - 2022-05-24

Fixed

conn busy error when querying Tasks

[0.13.1] - 2022-05-24

Fixed

conn busy error when querying Algos

[0.13.0] - 2022-05-23

Fixed

In standalone mode, truncate TimeService time to microsecond resolution to match PostgreSQL timestamp resolution.

Changed

Disable CGO.
More validation of Algo inputs (data managers / data samples)

Added

Introduce compute task inputs existing tasks won't have any inputs
Embed historical assets in the event messages.

[0.12.0] - 2022-05-16

Added

New mandatory name field to compute plan

Changed

Remove event column

[0.11.0] - 2022-05-09

Added

Add a new ALGO_PREDICT algo category

Changed

Validate algo inputs and outputs

[0.10.0] - 2022-05-03

Changed

Remove model asset column
Remove performance asset column
Remove datamanager asset column
Remove datasample asset column
Algos now have Inputs and Outputs
The orchestrator-server doesn't run DB migrations on startup anymore

Removed

ASSET_METRIC kind

[0.9.2] - 2022-04-15

Changed

Build with go 1.18

Fixed

Update failure report asset column migration to prevent null value error when migrating a populated database
Parent tasks keys format validation

[0.9.1] - 2022-04-13

Fixed

Order parent tasks keys by task position

[0.9.0] - 2022-04-13

Added

Added ALGO_METRICS Algo category

Changed

Remove compute task asset column
QueryAlgos filter "Category" is now "Categories"
Remove failure report asset column

Removed

Metrics gRPC routes. Use Algo gRPC routes and ALGO_METRICS category instead.

[0.8.0] - 2022-04-11

Added

Allow querying datasamples by keys

Changed

Remove node asset column
Remove algo asset column
Remove compute plan asset column

Fixed

Do not panic on nil filter

[0.7.0] - 2022-03-29

Added

Expose gRPC metrics
Expose database transaction and events metrics
Expose task metrics

Changed

Log SQL errors regardless of log level
Remove asset column of nodes table

Fixed

Publish events sequentially, preserving the order

[0.6.1] - 2022-03-01

Added

add support for graceful shutdown on SIGTERM signal

Changed

removed codegen layer and implicit protojson serialization

Fixed

Cancel all tasks when cancelling a compute plan
Check for compute plan existence on task registration
Disallow registration of tasks on a compute plan you don't own

[0.6.0] - 2022-02-18

Added

add Start and End timestamp filters for EventQueryFilter
support composite tasks with two composite parents
Add migration logs
add owner field to failure report asset
Add a new endpoint to register multiple models at the same time

Changed

return datasamples list in RegisterDataSamplesResponse
return tasks list in RegisterTasksResponse
store the error type of a failed compute task in a failure report instead of an event
improve performance of compute_tasks SQL indexes by using dedicated columns instead of JSONB
improve performance of compute plan queries by leveraging a specific index for status count
isolation level of read-only queries in standalone mode is now READ COMMITTED
improve performance of model SQL indexes by using dedicated columns instead of JSONB

Fixed

set the correct name of the RegisterFailureReport service method used in distributed mode
Return the correct models in GetComputeTaskInputModels for composite tasks
timestamp comparison when performing event sorting and filtering in PostgreSQL
ComputePlan query now uses correct SQL indexes
Incorrect sort order when checking parent task compatibility

Deprecated

RegisterModel gRPC method

[0.5.0] - 2022-01-16

Added

add a logs_permission field to the Dataset asset
add a GetDataSample method to the DataSample service

[0.4.0] - 2022-01-05

Added

add filter for compute plan query
chaincode now properly propagate request ID in every logs
log events as JSON
add FailureReport asset to store compute task failure information

[0.3.0] - 2021-11-30

Added

sort queried events
expose basic metrics from server, chaincode and forwarder behind METRICS_ENABLED feature flag
filter queried events on metadata

[0.2.0] - 2021-11-02

Changed

(BREAKING) Replace objective by metric
(BREAKING) Multiple metrics and performances per test task
fail gRPC healthcheck and stop serving on message broker disconnection

Added

Get task counts grouped by status when querying compute plans

Fixed

Events queried from the gRPC API now have their channel properly set
Leverage asset_key index when querying events

[0.1.0] - 2021-10-04

Fixed

Stable sorting of tasks

Added

Expose the orchestrator version and chaincode version

[0.0.2] - 2021-09-16

Added

Expose worker in task event metadata
Assets expose a creation date
Query algo by compute plan
Handle event backlog
Retry on fabric timeout
Add request ID to log context

Changed

Do not retry on assets out of sync
Do not compute plan status on model deletion
Reuse gateway connection in distributed mode
Replace readinessProbe by startupProbe
Do not cascade canceled status

Fixed

Properly retry on postgres' serialization error
Filtering events by asset in distributed mode
Input models for composite child of aggregate

[0.0.1] - 2021-06-29

Added

Automatic generation of graphviz documentation from *.proto file definition
asset management
asset event dispatch
standalone database (postgresql) support
distributed ledger (hyperledger-fabric) support