v2 control plane grpc + various fixes for scalability (#229)

* add transport settings for v2client * add error codes * fix httpmock tests * add model to cache if failed unload * add error handling for unload model * tidy up logic * add test * lint * limit http / grpc connections. * increase http connection timeout * Tidy up tests * lint * fix test * Update server snapshot creation in scheduler * read events in go routines * add v2 control plane protos * initial v2 grpc impl * add counter for load / unload models * add test * compute counts without models * add triton control plance to v2 dataplane for now * protos * compute memory including loading models * fix scheduler test * fix test * add a note * create a copy * cap * add a mutex around state map management * do not access map directly in memory store * return err * remove sleep * refactor and fix test * add v2 grpc test * add more test cases * add aggregate infer metrics and test * fix test and lint Co-authored-by: Clive Cox <cc@seldon.io>
SeldonIO · May 25, 2022 · e3f98cc · e3f98cc
1 parent 25246b7
commit e3f98cc
Show file tree

Hide file tree

Showing 26 changed files with 1,748 additions and 181 deletions.
diff --git a/apis/Makefile b/apis/Makefile
@@ -17,7 +17,8 @@ build:
 		./mlops/scheduler/scheduler.proto \
 		./mlops/scheduler/storage.proto \
 		./mlops/chainer/chainer.proto \
-		./mlops/v2_dataplane/v2_dataplane.proto
+		./mlops/v2_dataplane/v2_dataplane.proto \
+		./mlops/v2_model_repository/v2_model_repository.proto
 
 
 PROTOC_PYTHON_OUT := --python_out=./python

diff --git a/apis/mlops/agent/agent.pb.go b/apis/mlops/agent/agent.pb.go
diff --git a/apis/mlops/agent_debug/agent_debug.pb.go b/apis/mlops/agent_debug/agent_debug.pb.go
diff --git a/apis/mlops/chainer/chainer.pb.go b/apis/mlops/chainer/chainer.pb.go
diff --git a/apis/mlops/proxy/proxy.pb.go b/apis/mlops/proxy/proxy.pb.go
diff --git a/apis/mlops/scheduler/scheduler.pb.go b/apis/mlops/scheduler/scheduler.pb.go