Services

Until now, dstack has supported dev-environment and task as configuration types. With the 0.10.7 update, we introduce
service, a dedicated configuration type for serving.

Usage example:

type: service

gateway: ${{ secrets.GATEWAY_ADDRESS }}

image: ghcr.io/huggingface/text-generation-inference:0.9.3

port: 8000

commands: 
  - text-generation-launcher --hostname 0.0.0.0 --port 8000 --trust-remote-code

The gateway property represents the address of a special cloud instance that wraps the running service with a public endpoint.

Gateways

Before you can run a service, you have to configure a gateway.

First, you have to create a gateway in a project of your choice using the dstack gateway create command:

dstack gateway create

Once the gateway is up, the command will print its address. Go ahead and create a secret with this address.

dstack secrets add GATEWAY_ADDRESS <gateway address>

That's it! Now you can run your service using the dstack run command, which deploys the service and forwards the traffic to the gateway, thereby providing you with a public endpoint.

This initial support for services is the first step towards providing multi-cloud and cost-effective inference. In the near future, we plan to make it more functional and easier to use.

What's changed

• Support restarting local and GCP runs by @r4victor in #590
• Add API for listing all runs by @r4victor in #591
• The dstack init should handle invalid Git credentials by @peterschmidt85 in #594
• Support custom run names by @r4victor in #595
• dstack doesn't work if the repo contains a submodule with SSH URL by @peterschmidt85 in #598
• Always use cuda images for instances with GPU by @r4victor in #602
• Introduce gateways for services publication by @Egor-S in #596
• Small ports refactoring by @Egor-S in #603
• #588 Created All run list page. by @olgenn in #607
• Improve gateway security, show verbose errors by @Egor-S in #608
• Do not require sshd in task configuration for custom docker images by @Egor-S in #609
• #588 all runs list by @olgenn in #611
• [Bug]: Doesn't run a dev environment if code is not configured in PATH by @peterschmidt85 in #613
• Allow to create gateways in AWS and Azure by @Egor-S in #614
• [Bug]: dstack init doesn't work if .git/config or ~/.gitconfig doesn't have the user section by @peterschmidt85 in #617
• [Bug]: dstack config doesn't work if ~/.dstack doesn't exist by @peterschmidt85 in #618
• Migrate to Gen2 images for Azure, add A100 support by @Egor-S in #619
• Add time.Sleep() in /logsws handler by @r4victor in #623
• Services & gateway docs by @Egor-S in #620
• Replace localhost with gateway hostname in service logs by @Egor-S in #624

Changelog: 0.10.6...0.10.7