Add an example showing deployment using TensorFlow Serving and Kubernetes #453

sayakpaul · 2022-08-07T13:15:39Z

add: initial files for the blog.

sayakpaul · 2022-08-07T13:18:30Z

_blog.yml

+  title: "Deploying 🤗 ViT on Kubernetes with TF Serving"
+  authors: chansung, sayakpaul
+  thumbnail: /blog/assets/86_nystromformer/thumbnail.png
+  date: August 15, 2022


deploy-tfserving-kubernetes.md

merveenoyan

Thanks a lot for this blog post, I refreshed my GCP knowledge while reviewing! I left couple of suggestions and nits 🙂

deploy-tfserving-kubernetes.md

merveenoyan · 2022-08-08T14:03:41Z

deploy-tfserving-kubernetes.md

+Therefore, we assume some familiarity with Docker and Kubernetes.
+
+This post builds on top of the [<u>previous post</u>](https://huggingface.co/blog/tf-serving-vision). So, we highly
+recommend reading it if not already done. You can find all the code


Suggested change

recommend reading it if not already done. You can find all the code

recommend reading it if it's not already read. You can find all the code

Not too sure about this passive voicing.

deploy-tfserving-kubernetes.md

merveenoyan · 2022-08-08T14:44:18Z

deploy-tfserving-kubernetes.md

+import base64
+
+image_path = tf.keras.utils.get_file(
+    "image.jpg", "http://images.cocodataset.org/val2017/000000039769.jpg"


maybe mention later they can get it from bucket URI as well (I think it's cool for the sake of being end-to-end here)

Could you elaborate?

I think it depends on who is the end-user. For service consumers from personal devices, it is common to make requests with data from arbitrary sources. However, if TF Serving is just a part of micro service architecture, then it would make sense to say "end-to-end" with GCS bucket URI.

Maybe we could just a leave a short comment like you can get a file from GCS with get_file API as well.

I don't think get_file() supports gs:// URIs.

I tried the following (video-api-storage bucket has public viewing enabled):

video_gcs_uri = "gs://video-api-storage/sample_video.mp4" path_to_downloaded_file = tf.keras.utils.get_file("video.mp4", video_gcs_uri)

It results it:

Exception Traceback (most recent call last) [/usr/local/lib/python3.7/dist-packages/keras/utils/data_utils.py](https://localhost:8080/#) in get_file(fname, origin, untar, md5_hash, file_hash, cache_subdir, hash_algorithm, extract, archive_format, cache_dir) 279 raise Exception(error_msg.format(origin, e.code, e.msg)) 280 except urllib.error.URLError as e: --> 281 raise Exception(error_msg.format(origin, e.errno, e.reason)) 282 except (Exception, KeyboardInterrupt) as e: 283 if os.path.exists(fpath): Exception: URL fetch failure on gs://video-api-storage/sample_video.mp4: None -- unknown url type: gs

From the source code, we can see that it doesn't have any GCS specific utilities built-in.

merveenoyan · 2022-08-08T14:45:44Z

deploy-tfserving-kubernetes.md

+various options to tailor the deployment based on your application use
+case. Below, we briefly discuss some of them.
+
+**`enable_batching`** enables the batch inference capability that


Batching in cloud is particularly useful if you don't want to pay continuously (I think this is a super cool info to mention to people can save 💴 💵 ) but it's your choice 😅

Once we have an instance of TFServing, people could choose where to deploy it. So we left just a general ideas about batching capability. We will discuss more in-depth in the next blog posts.

However, I really appreciate your comment since I have not thought about it as a money saver :)

It's a money saver only when you can afford to run things in an offline mode. We can definitely mention that in a bit more detail.

Added a note.

deploy-tfserving-kubernetes.md

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

sayakpaul · 2022-08-09T02:56:05Z

@merveenoyan @deep-diver addressed almost all the PR comments. PTAL.

osanseviero

Cool blog post! Two suggestions:

Consistently capitalize Pods. It's somewhat inconsistent at the moment.
Try to be a bit more consistent between first and second person. Not a big issue though!

Thanks a lot for the contribution 🔥 it's amazing! 🚀

_blog.yml

deploy-tfserving-kubernetes.md

sayakpaul · 2022-08-09T09:19:53Z

Try to be a bit more consistent between first and second person. Not a big issue though!

You'd notice "you" and "we" both. Yes, we're aware of it.

If you look into it a bit more deeply, you will notice that "we" is used quite sparingly and in cases where we deliberately want to highlight something about us rather than deflecting the reader.

In all the other cases, we have tried maintaining "you" throughout.

With this in mind, if you spot any inconsistencies regarding this, please let us know.

Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

deep-diver · 2022-08-09T12:58:01Z

@osanseviero @sayakpaul

addressed the comments

sayakpaul · 2022-08-09T14:50:47Z

deploy-tfserving-kubernetes.md

+We applied the same deployment workflow for an ONNX-optimized version of the same
+Vision Transformer model. For more details, check out [this link](https://github.com/sayakpaul/deploy-hf-tf-vision-models/tree/main/hf_vision_model_onnx_gke). ONNX-optimized models are especially beneficial if you're using x86 CPUs for deployment.


@osanseviero @merveenoyan added a note regarding ONNX deployments since many HF users are already familiar with it. Our repository supports deploying an ONNX-optimized version of the same ViT model to a Kubernetes cluster :)

Cc: @deep-diver

deep-diver · 2022-08-09T23:14:39Z

we need an approval from @merveenoyan :) more suggestions?

merveenoyan

LGTM! But I'd like @stevhliu to have a final look to see if there's anything. @stevhliu you can also merge given Omar and me will be gone next week. (Omar is already off)

stevhliu

Super interesting read, especially for someone who doesn't know too much about Docker/Kubernetes! 🤗

deploy-tfserving-kubernetes.md

stevhliu · 2022-08-11T15:40:53Z

deploy-tfserving-kubernetes.md

+
+In this section, you’ll see how to containerize that model using the
+[<u>base TensorFlow Serving Image</u>](http://hub.docker.com/r/tensorflow/serving/tags/). TensorFlow Serving consumes models
+in the [`SavedModel`](https://www.tensorflow.org/guide/saved_model) format. Recall how you


Same comment as above about having the reader refer to another post.

Maybe keep this one to empathize with the reader in case they feel clueless about the SavedModel?

Sure, that's a good compromise :)

deploy-tfserving-kubernetes.md

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

sayakpaul · 2022-08-11T16:23:57Z

@stevhliu all comments addressed :)

sayakpaul added 3 commits August 7, 2022 11:12

Empty-Commit

692e987

add: initial files for the blog.

422a3af

Merge pull request #1 from sayakpaul/dev/tfserving-kubernetes

a626c27

add: initial files for the blog.

sayakpaul commented Aug 7, 2022

View reviewed changes

deploy-tfserving-kubernetes.md Outdated Show resolved Hide resolved

update thumbnail image

2212b38

osanseviero requested review from merveenoyan and osanseviero August 8, 2022 06:54

merveenoyan requested changes Aug 8, 2022

View reviewed changes

sayakpaul and others added 9 commits August 9, 2022 07:47

Update deploy-tfserving-kubernetes.md

02e260d

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

f25a7b0

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

5e5bd9c

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

ad07986

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

de1bc8b

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

613f22e

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

0801c89

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

Update deploy-tfserving-kubernetes.md

cf3c4ed

Co-authored-by: Merve Noyan <merveenoyan@gmail.com>

address PR comments.

e80f998

sayakpaul requested a review from merveenoyan August 9, 2022 02:56

osanseviero approved these changes Aug 9, 2022

View reviewed changes

deep-diver and others added 5 commits August 9, 2022 21:43

Update deploy-tfserving-kubernetes.md

3e82998

Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

Update deploy-tfserving-kubernetes.md

1b6034c

Co-authored-by: Omar Sanseviero <osanseviero@gmail.com>

capitalize pods to Pods

a637d8a

update author-card href

c50d5a5

include github repo in conclusion

eb9e97f

sayakpaul added 2 commits August 9, 2022 19:05

fix: _blog.yml

07fda36

chore: add note on the onnx deployment.

362165a

adjust line breaks.

8f08aed

sayakpaul commented Aug 9, 2022

View reviewed changes

merveenoyan approved these changes Aug 11, 2022

View reviewed changes

Merge branch 'main' into add/tfserving-kubernetes

24534d8

stevhliu reviewed Aug 11, 2022

View reviewed changes

sayakpaul and others added 14 commits August 11, 2022 21:40

Update deploy-tfserving-kubernetes.md

0c00e20

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

2731e3e

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

9d9ae26

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

5d6d07f

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

967eedb

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

16f7349

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

5971654

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

73bb2f8

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

3dbdd3f

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

c9d9e59

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

72144b5

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

51f0700

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

41ab0fd

Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com>

Update deploy-tfserving-kubernetes.md

ca9df66

stevhliu merged commit e9df715 into huggingface:main Aug 11, 2022

sayakpaul deleted the add/tfserving-kubernetes branch August 11, 2022 16:31

	recommend reading it if not already done. You can find all the code
	recommend reading it if it's not already read. You can find all the code

		We applied the same deployment workflow for an ONNX-optimized version of the same
		Vision Transformer model. For more details, check out [this link](https://github.com/sayakpaul/deploy-hf-tf-vision-models/tree/main/hf_vision_model_onnx_gke). ONNX-optimized models are especially beneficial if you're using x86 CPUs for deployment.

Add an example showing deployment using TensorFlow Serving and Kubernetes #453

Add an example showing deployment using TensorFlow Serving and Kubernetes #453

Uh oh!

Conversation

sayakpaul commented Aug 7, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

merveenoyan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sayakpaul Aug 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sayakpaul commented Aug 9, 2022

Uh oh!

osanseviero left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul commented Aug 9, 2022

Uh oh!

deep-diver commented Aug 9, 2022

Uh oh!

Choose a reason for hiding this comment

Uh oh!

deep-diver commented Aug 9, 2022

Uh oh!

merveenoyan left a comment

Choose a reason for hiding this comment

Uh oh!

stevhliu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sayakpaul Aug 9, 2022 •

edited

Loading