Skip to content

Latest commit

 

History

History
51 lines (33 loc) · 2.03 KB

README.md

File metadata and controls

51 lines (33 loc) · 2.03 KB
Raw

gRPC multiprocessing with streaming

This repo is a boilerplate showing how you can use multiprocessing with gRPC with streaming capabilities in Python

Payloads streaming allows bypassing gRPC's 2Gb limit in its Python implementation.

Getting started

This project offers a minimum reproducible example of clients sending batches of videos to gRPC servers.

  • Input : video(s)
  • Output : thumbnail(s)

client.py starts a pool of several gRPC channels. Each client process is attributed a video and cuts it into chunks that are sent to one of the gRPC servers parallely to other client processes.

Each response including the video thumbnail gets returned and cut if exceeding 2Gb as well.

  1. Adjust the NUM_WORKERS and NUM_JOBS variables in docker-compose.yml if desired

  2. Add your videos in the input/ directory. Thumbnails will be saved in output/.

# Download 2 videos (to test multiprocessing)
wget -O ./input/video.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
wget -O ./input/video_2.mp4 https://file-examples-com.github.io/uploads/2017/04/file_example_MP4_640_3MG.mp4
  1. Start playing around by running :
docker-compose build
docker-compose run client

Re-compile proto

You must have python3 and pip installed

# In project root directory
python3 -m pip install grpcio==1.32.0 protobuf==3.12.2
python3 -m grpc_tools.protoc --proto_path=./grpc_config --python_out=./grpc_config --grpc_python_out=./grpc_config ./grpc_config/video_thumbnail.proto

Credits