Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add grpc router #90

Merged
merged 12 commits into from
Nov 28, 2023
Merged

feat: add grpc router #90

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
ae86f71
add watcher
OlivierDehaene Nov 23, 2023
b856ec7
refacto cors
OlivierDehaene Nov 23, 2023
85b0990
move prometheus builder
OlivierDehaene Nov 23, 2023
113df89
grpc support
OlivierDehaene Nov 23, 2023
94919a2
add images
OlivierDehaene Nov 24, 2023
cb2eeab
fix openapi
OlivierDehaene Nov 24, 2023
dbb232b
add unzip
OlivierDehaene Nov 24, 2023
1d61277
add proto
OlivierDehaene Nov 24, 2023
619ecbf
multiple services
OlivierDehaene Nov 25, 2023
858bb4f
prepare for streaming
OlivierDehaene Nov 25, 2023
b8e8f20
stream
OlivierDehaene Nov 27, 2023
89392b0
preserve order streams
OlivierDehaene Nov 27, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 35 additions & 1 deletion .github/workflows/build_75.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@
tags: |
type=semver,pattern=turing-{{version}}
type=semver,pattern=turing-{{major}}.{{minor}}
type=raw,value=turing-latest,enable=${{ github.ref == format('refs/heads/{0}', github.event.repository.default_branch) }}
type=raw,value=turing-latest
type=raw,value=turing-sha-${{ env.GITHUB_SHA_SHORT }}
- name: Build and push Docker image
id: build-and-push-75
Expand All @@ -99,3 +99,37 @@
labels: ${{ steps.meta-75.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-75,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-75,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-75-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern=turing-{{version}}+grpc
type=semver,pattern=turing-{{major}}.{{minor}}+grpc
type=raw,value=turing-latest+grpc
type=raw,value=turing-sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-75-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile-cuda
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
CUDA_COMPUTE_CAP=75
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
DEFAULT_USE_FLASH_ATTENTION=False
tags: ${{ steps.meta-75-grpc.outputs.tags }}
labels: ${{ steps.meta-75-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-75,mode=max
33 changes: 33 additions & 0 deletions .github/workflows/build_80.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,36 @@
labels: ${{ steps.meta-80.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-80,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-80,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-80-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern={{version}}+grpc
type=semver,pattern={{major}}.{{minor}}+grpc
type=raw,value=latest+grpc
type=raw,value=sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-80-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile-cuda
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
CUDA_COMPUTE_CAP=80
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
tags: ${{ steps.meta-80-grpc.outputs.tags }}
labels: ${{ steps.meta-80-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-80,mode=max
34 changes: 34 additions & 0 deletions .github/workflows/build_86.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,37 @@
labels: ${{ steps.meta-86.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-86,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-86,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-86-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern=86-{{version}}+grpc
type=semver,pattern=86-{{major}}.{{minor}}+grpc
type=raw,value=86-latest+grpc
type=raw,value=86-sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-86-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile-cuda
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
CUDA_COMPUTE_CAP=86
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
tags: ${{ steps.meta-86-grpc.outputs.tags }}
labels: ${{ steps.meta-86-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-86,mode=max

34 changes: 34 additions & 0 deletions .github/workflows/build_89.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,3 +98,37 @@
labels: ${{ steps.meta-89.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-89,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-89,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-89-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern=89-{{version}}+grpc
type=semver,pattern=89-{{major}}.{{minor}}+grpc
type=raw,value=89-latest+grpc
type=raw,value=89-sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-89-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile-cuda
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
CUDA_COMPUTE_CAP=89
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
tags: ${{ steps.meta-89-grpc.outputs.tags }}
labels: ${{ steps.meta-89-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-89,mode=max

35 changes: 35 additions & 0 deletions .github/workflows/build_90.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,4 +98,39 @@
labels: ${{ steps.meta-90.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-90,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-90,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-90-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern=hopper-{{version}}+grpc
type=semver,pattern=hopper-{{major}}.{{minor}}+grpc
type=raw,value=hopper-latest+grpc
type=raw,value=hopper-sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-90-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile-cuda
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
CUDA_COMPUTE_CAP=90
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
tags: ${{ steps.meta-90-grpc.outputs.tags }}
labels: ${{ steps.meta-90-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-90,mode=max



33 changes: 33 additions & 0 deletions .github/workflows/build_cpu.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,3 +97,36 @@
labels: ${{ steps.meta-cpu.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-cpu,mode=max
cache-to: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-cpu,mode=max
- name: Extract metadata (tags, labels) for Docker
id: meta-cpu-grpc
uses: docker/metadata-action@v4.3.0
with:
images: |
registry.internal.huggingface.tech/api-inference/text-embeddings-inference
ghcr.io/huggingface/text-embeddings-inference
flavor: |
latest=false
tags: |
type=semver,pattern=cpu-{{version}}+grpc
type=semver,pattern=cpu-{{major}}.{{minor}}+grpc
type=raw,value=cpu-latest+grpc
type=raw,value=cpu-sha-${{ env.GITHUB_SHA_SHORT }}+grpc
- name: Build and push Docker image
id: build-and-push-cpu-grpc
uses: docker/build-push-action@v4
with:
context: .
target: grpc
file: Dockerfile
push: ${{ github.event_name != 'pull_request' }}
platforms: 'linux/amd64'
build-args: |
SCCACHE_GHA_ENABLED=on
ACTIONS_CACHE_URL=${{ env.ACTIONS_CACHE_URL }}
ACTIONS_RUNTIME_TOKEN=${{ env.ACTIONS_RUNTIME_TOKEN }}
GIT_SHA=${{ env.GITHUB_SHA }}
DOCKER_LABEL=sha-${{ env.GITHUB_SHA_SHORT }}
tags: ${{ steps.meta-cpu-grpc.outputs.tags }}
labels: ${{ steps.meta-cpu-grpc.outputs.labels }}
cache-from: type=registry,ref=registry.internal.huggingface.tech/api-inference/text-embeddings-inference:cache-cpu,mode=max

Loading
Loading