Skip to content

v0.10.0-alpha

Pre-release
Pre-release
Compare
Choose a tag to compare
@droplet-bot droplet-bot released this 02 Jun 21:20
v0.10.0-alpha
8df621b
This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
GPG key ID: 4AEE18F83AFDEB23
Expired
Learn about vigilant mode.

🎉 Highlights

New pipeline recipe, telemetry integrations and license updates

Please refer to the following sections for more details.

🅱️ [Breaking change] New pipeline recipe

In this release, we have adopted a new pipeline recipe format (breaking change) to lay the foundation to support more complex and flexible pipelines in the future.
A recipe describes a pipeline resource consisting of components including:

  • source component - where the pipeline starts to ingest unstructured data to be processed
  • model components - several deployed AI models to process the ingested unstructured data in parallel
  • destination components - several destinations to send the processed outputs in parallel
{
  "version": "v1alpha",
  "components": [
    {
      "id": <component-id>,
      "resource_name": <source-connector-resource-name>
    },
    {
      "id": <component-id>,
      "resource_name": <model-resource-name>
    },
    ...
    {
      "id": <component-id>,
      "resource_name": <destination-connector-resource-name>
    }
  ]
}

👉 Check out the documentation here

🅱️ [Breaking change] Trigger endpoint updates

The old triggering endpoint /trigger was retired and replaced with fine-grained endpoints based on the pipeline mode:

  • the trigger endpoint for a SYNC pipeline is /triggerSync
  • the trigger endpoint for an ASYNC pipeline /triggerAsync

New telemetry integrations

Observability is critical for distributed microservice architecture. Through OpenTelemetry, we can generate, collect and export metrics, logs and traces to help analyse VDP’s performance and behaviour.

How to enable the VDP observability stack?

You can enable the observability stack by setting ENABLE_OBSERVE=true in the .env file. The following telemetry tools are supported now.

  • Jaeger (localhost:16686): OpenTelemetry allows us to export spans to Jaeger. Use Jaeger when you want to debug the complete flow of a request through the VDP services.
  • InfluxDB (localhost: 8086, username:admin, password:password): detailed metrics are sent to InfluxDB for monitoring, and are imported into the Grafana dashboard
  • Grafana (localhost:3002, username:admin, password:admin): the Grafana dashboard visualises the metrics to monitor the performance and anomalies of VDP services
  • Prometheus (localhost:9090): VDP exports metrics like vdp_pipeline_sync_trigger_counter_total (total number of triggers from SYNC pipelines) and vdp_pipeline_async_trigger_counter_total (total number of triggers from ASYNC pipelines) to Prometheus.

observability-integration

License updates

From this release, VDP will adopt multiple licenses.

  • VDP Protocol: open-source and available under the MIT License
  • VDP Connectors (including data source connectors, data destination connectors and model connectors): open-source and available under the MIT License
  • UI, API and services of VDP: source available under the Elastic License 2.0 (ELv2)
  • Instill Cloud (fully-managed public cloud service for VDP): closed-source

diagram-export-6_4_2023, 12_39_43 AM

Why are VDP connectors open source?

We want to build VDP as an open platform that integrates with other fantastic tooling and services. And the best way to build these integrations is via open source, whether it is a data connector connecting data to a warehouse, application, or database, or a model connector connecting models from self-hosted or 3rd party platforms. This also benefits the projects that want to integrate with these connectors.

Why did VDP services adopt ELv2?

Our mission is to make AI accessible to everyone. The best way to achieve this is to make VDP free to use and source available to everyone, while ensuring we safely create a sustainable business. Therefore, we chose ELv2 since it is very permissive. On the one hand, this allows us to provide users with free access to our source code, the permission to modify it. On the other hand, we've released Instill Cloud, a fully-managed public cloud service built upon VDP that offers additional features. With the license, we don't have to worry about our project being taken by some other companies for monetization.

👉 Check the License FAQ here.

Console 0.23.0-alpha

Features

  • [INS-518] update dummy model for testing (#419) (b6a3745)
  • [INS-751] adapt new pipeline recipe and update trigger snippet (#417) (61809bb)
  • stabilize test related to the pipeline details page (#414) (9f72427)
  • update docs link at sidebar (#416) (51103f9)

Bug Fixes

  • [INS-805] fix not correctly query pipelines that use this resource (#420) (bdcd7aa)
Assets 2