Skip to content

Make gateway active#829

Merged
Egor-S merged 44 commits intomasterfrom
issue-799-gateway-app
Jan 15, 2024
Merged

Make gateway active#829
Egor-S merged 44 commits intomasterfrom
issue-799-gateway-app

Conversation

@Egor-S
Copy link
Copy Markdown
Contributor

@Egor-S Egor-S commented Jan 9, 2024

This PR changes the communication scheme between the server, job instance, and gateway.

Closes #799

  • The gateway now runs the FastAPI app as a systemd service
  • The gateway now establishes an SSH tunnel to the job instance (contrary to the previous behavior)
  • The server registers and unregisters a service via secured HTTP requests to the gateway
  • The gateway features an OpenAI interface
    • OpenAI requests are converted to TGI format and sent to the job instance
    • OpenAI interface supports /models query (scoped by the project)
  • If run from source, the server picks the latest build version related to the HEAD

Steps to use an OpenAI interface:

  1. Create a new default gateway
  2. Redirect the DNS record to the gateway
  3. Add the following to the service configuration (TGI)
    model:
      type: chat
      name: TheBloke/Mistral-7B-OpenOrca-AWQ
      format: tgi
      # optionally override `chat_template:` and `eos_token:` here
    
  4. Run the service using dstack run
  5. Make a request to gateway.<your domain> using openai python package

TODO before merge:

  • Update gateway app on server start up
  • Build gateway app in release workflow

@Egor-S Egor-S merged commit fb92600 into master Jan 15, 2024
@Egor-S Egor-S deleted the issue-799-gateway-app branch January 16, 2024 12:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Make gateway active

2 participants