Fix http server bug (#1096)

* refactoring(create_table done) * refactoring * refactor server delivery (insert done) * refactoring server module (count_table done) * server refactor done * cmake pass * refactor server module done. * set grpc response status correctly * format done. * fix redefine ErrorMap() * optimize insert reducing ids data copy * optimize grpc request with reducing data copy * clang format * [skip ci] Refactor server module done. update changlog. prepare for PR * remove explicit and change int32_t to int64_t * add web server * [skip ci] add license in web module * modify header include & comment oatpp environment config * add port configure & create table in handler * modify web url * simple url complation done & add swagger * make sure web url * web functionality done. debuging * add web unittest * web test pass * add web server port * add web server port in template * update unittest cmake file * change web server default port to 19121 * rename method in web module & unittest pass * add search case in unittest for web module * rename some variables * fix bug * unittest pass * web prepare * fix cmd bug(check server status) * update changlog * add web port validate & default set * clang-format pass * add web port test in unittest * add CORS & redirect root to swagger ui * add web status * web table method func cascade test pass * add config url in web module * modify thirdparty cmake to avoid building oatpp test * clang format * update changlog * add constants in web module * reserve Config.cpp * fix constants reference bug * replace web server with async module * modify component to support async * format * developing controller & add test clent into unittest * add web port into demo/server_config * modify thirdparty cmake to allow build test * remove unnecessary comment * add endpoint info in controller * finish web test(bug here) * clang format * add web test cpp to lint exclusions * check null field in GetConfig * add macro RETURN STATUS DTo * fix cmake conflict * fix crash when exit server * remove surplus comments & add http param check * add uri /docs to direct swagger * format * change cmd to system * add default value & unittest in web module * add macros to judge if GPU supported * add macros in unit & add default in index dto & print error message when bind http port fail * format (fix #788) * fix cors bug (not completed) * comment cors * change web framework to simple api * comments optimize * change to simple API * remove comments in controller.hpp * remove EP_COMMON_CMAKE_ARGS in oatpp and oatpp-swagger * add ep cmake args to sqlite * clang-format * change a format * test pass * change name to * fix compiler issue(oatpp-swagger depend on oatpp) * add & in start_server.h * specify lib location with oatpp and oatpp-swagger * add comments * add swagger definition * [skip ci] change http method options status code * remove oatpp swagger(fix #970) * remove comments * check Start web behavior * add default to cpu_cache_capacity * remove swagger component.hpp & /docs url * remove /docs info * remove /docs in unittest * remove space in test rpc * remove repeate info in CHANGLOG * change cache_insert_data default value as a constant * [skip ci] Fix some broken links (#960) * [skip ci] Fix broken link * [skip ci] Fix broken link * [skip ci] Fix broken link * [skip ci] Fix broken links * fix issue 373 (#964) * fix issue 373 * Adjustment format * Adjustment format * Adjustment format * change readme * #966 update NOTICE.md (#967) * remove comments * check Start web behavior * add default to cpu_cache_capacity * remove swagger component.hpp & /docs url * remove /docs info * remove /docs in unittest * remove space in test rpc * remove repeate info in CHANGLOG * change cache_insert_data default value as a constant * adjust web port cofig place * rename web_port variable * change gpu resources invoke way to cmd() * set advanced config name add DEFAULT * change config setting to cmd * modify .. * optimize code * assign TableDto' count default value 0 (fix #995) * check if table exists when show partitions (fix #1028) * check table exists when drop partition (fix #1029) * check if partition name is legal (fix #1022) * modify status code when partition tag is illegal * update changlog * add info to /system url * add binary index and add bin uri & handler method(not completed) * optimize http insert and search time(fix #1066) | add binary vectors support(fix #1067) * fix test partition bug * fix test bug when check insert records * add binary vectors test * add default for offset and page_size * fix uinttest bug * [skip ci] remove comments * optimize web code for PR comments * add new folder named utils * check offset and pagesize (fix #1082) * improve error message if offset or page_size is not legal (fix #1075) * add log into web module * update changlog * check gpu sources setting when assign repeated value (fix #990) * update changlog * clang-format pass * add default handler in http handler * [skip ci] improve error msg when check gpu resources * change check offset way * remove func IsIntStr * add case * change int32 to int64 when check number str * add log in we module(doing) * update test case * add log in web controller Co-authored-by: jielinxu <52057195+jielinxu@users.noreply.github.com> Co-authored-by: JackLCL <53512883+JackLCL@users.noreply.github.com> Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
milvus-io · Jan 19, 2020 · 6151733 · 6151733
1 parent 658f100
commit 6151733
Show file tree

Hide file tree

Showing 10 changed files with 294 additions and 106 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,13 +11,16 @@ Please mark all change in change log and use the issue from GitHub
 - \#805 - IVFTest.gpu_seal_test unittest failed
 - \#831 - Judge branch error in CommonUtil.cpp
 - \#977 - Server crash when create tables concurrently
+- \#990 - check gpu resources setting when assign repeated value
 - \#995 - table count set to 0 if no tables found
 - \#1010 - improve error message when offset or page_size is equal 0
 - \#1022 - check if partition name is legal
 - \#1028 - check if table exists when show partitions
 - \#1029 - check if table exists when try to delete partition
 - \#1066 - optimize http insert and search speed
 - \#1067 - Add binary vectors support in http server
+- \#1075 - improve error message when page size or offset is illegal
+- \#1082 - check page_size or offset value to avoid float
 
 ## Feature
 - \#216 - Add CLI to get server info

diff --git a/core/src/server/Config.cpp b/core/src/server/Config.cpp
@@ -22,6 +22,7 @@
 #include <regex>
 #include <string>
 #include <unordered_map>
+#include <unordered_set>
 #include <vector>
 
 #include "config/YamlConfigMgr.h"
@@ -791,9 +792,19 @@ Config::CheckGpuResourceConfigSearchResources(const std::vector<std::string>& va
         return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
+    std::unordered_set<std::string> value_set;
     for (auto& resource : value) {
         CONFIG_CHECK(CheckGpuResource(resource));
+        value_set.insert(resource);
     }
+
+    if (value_set.size() != value.size()) {
+        std::string msg =
+            "Invalid gpu build search resource. "
+            "Possible reason: gpu_resource_config.gpu_search_resources contains duplicate resources.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
+    }
+
     return Status::OK();
 }
 
@@ -806,8 +817,17 @@ Config::CheckGpuResourceConfigBuildIndexResources(const std::vector<std::string>
         return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
+    std::unordered_set<std::string> value_set;
     for (auto& resource : value) {
         CONFIG_CHECK(CheckGpuResource(resource));
+        value_set.insert(resource);
+    }
+
+    if (value_set.size() != value.size()) {
+        std::string msg =
+            "Invalid gpu build index resource. "
+            "Possible reason: gpu_resource_config.build_index_resources contains duplicate resources.";
+        return Status(SERVER_INVALID_ARGUMENT, msg);
     }
 
     return Status::OK();

diff --git a/core/src/server/web_impl/Constants.h b/core/src/server/web_impl/Constants.h
@@ -21,6 +21,9 @@ namespace milvus {
 namespace server {
 namespace web {
 
+////////////////////////////////////////////////////
+static const char* WEB_LOG_PREFIX = "[Web] ";
+
 ////////////////////////////////////////////////////
 
 static const char* CORS_KEY_METHODS = "Access-Control-Allow-Methods";