Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add brpc serialization support. #11430

Merged
merged 136 commits into from
Dec 14, 2018
Merged
Show file tree
Hide file tree
Changes from 47 commits
Commits
Show all changes
136 commits
Select commit Hold shift + click to select a range
2acc420
add
gongweibao Jun 12, 2018
19cb300
add files
gongweibao Jun 12, 2018
b5eb369
add files
gongweibao Jun 12, 2018
68cd387
add serialize
gongweibao Jun 12, 2018
b3a6973
compile bserial
gongweibao Jun 13, 2018
9cd6f21
add brpc_serde_test
gongweibao Jun 13, 2018
b436597
fix
gongweibao Jun 13, 2018
fa0158a
add configure file
gongweibao Jun 13, 2018
10ad682
fix
gongweibao Jun 13, 2018
8e079b2
cleanup
gongweibao Jun 13, 2018
ddee50f
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jun 14, 2018
a52adb8
add sendcomplete
gongweibao Jun 14, 2018
3123576
fix send recv
gongweibao Jun 14, 2018
166a79f
brpc_server_test passed
gongweibao Jun 14, 2018
18e1743
fix grpc
gongweibao Jun 14, 2018
8888078
fix
gongweibao Jun 14, 2018
5032b93
fix
gongweibao Jun 14, 2018
baf17e2
fix
gongweibao Jun 14, 2018
1bf3783
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jun 14, 2018
4544329
add framework_proto
gongweibao Jun 14, 2018
8cc2fbe
fix complation error
gongweibao Jun 14, 2018
d7023bb
merge
gongweibao Jun 14, 2018
8c51dd1
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jun 14, 2018
f806963
merge
gongweibao Jun 14, 2018
2fe5b68
fix unittest
gongweibao Jun 14, 2018
0355764
add assert
gongweibao Jun 14, 2018
8780c08
clean up
gongweibao Jun 15, 2018
f8662a1
follow comments
gongweibao Jun 17, 2018
20bef4f
add grpc comments
gongweibao Jun 20, 2018
cb31718
fix brpc complation error!
gongweibao Jun 20, 2018
1f49a32
add openssl static lib
gongweibao Jun 20, 2018
1c5fe25
add openssl static lib
gongweibao Jun 20, 2018
c7dba8e
fix deps
gongweibao Jun 20, 2018
6c84d2c
fix cmake
gongweibao Jun 20, 2018
18b0274
fix cmake
gongweibao Jun 21, 2018
a6c7ae3
fix fetch barrier
gongweibao Jun 21, 2018
efe8abb
fix fetch bug
gongweibao Jun 21, 2018
b1c8a6d
add logs
gongweibao Jun 21, 2018
41f62a4
fix get error
gongweibao Jun 21, 2018
4ccd5b1
change 131072
gongweibao Jun 25, 2018
3e4fada
add zero copy
gongweibao Jun 25, 2018
a77340d
add
gongweibao Jun 25, 2018
66e2d44
clean up
gongweibao Jun 25, 2018
d2cd653
clean up
gongweibao Jun 25, 2018
12ab36d
merge
gongweibao Jun 26, 2018
f8f0fe0
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jun 26, 2018
3bb4c13
fix merge
gongweibao Jun 26, 2018
c1ebb7c
brpc cmake
gongweibao Jun 26, 2018
700c15b
add
gongweibao Jun 28, 2018
0680a27
add brpc rdma
gongweibao Jun 28, 2018
b2a5c94
fix compile
gongweibao Jun 28, 2018
b0fc28f
add
gongweibao Jun 28, 2018
1f97756
fix cmake
gongweibao Jun 29, 2018
1ee917e
merge
gongweibao Jun 29, 2018
aad877a
add fix
gongweibao Jun 29, 2018
a1a3899
add log
gongweibao Jun 29, 2018
ee3c7dd
fix local variable bug
gongweibao Jun 30, 2018
9965ac2
fix test
gongweibao Jun 30, 2018
f26322a
fix core
gongweibao Jun 30, 2018
0c2a211
rm openssl.cmake
gongweibao Jun 30, 2018
aa8a03a
fix bug
gongweibao Jul 2, 2018
6e109ac
add
gongweibao Jul 2, 2018
d5c5ec0
modify brpc branch
gongweibao Jul 5, 2018
f079aea
test
gongweibao Jul 9, 2018
65397b3
test dist
gongweibao Jul 9, 2018
1fb0793
merge
gongweibao Jul 9, 2018
a1a59db
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jul 13, 2018
30d9b11
fix rpc_client_t
gongweibao Jul 13, 2018
8f7904f
merge
gongweibao Jul 16, 2018
8aa933c
merge
gongweibao Jul 16, 2018
3834fa3
clean up
gongweibao Jul 16, 2018
b27585f
merge
gongweibao Jul 16, 2018
c52b365
Merge branch 'develop' of https://github.com/PaddlePaddle/Paddle into…
gongweibao Jul 17, 2018
d4665bb
fix config
gongweibao Jul 17, 2018
6fa5ac6
fix fluid benchmark
gongweibao Jul 20, 2018
22140d8
add
gongweibao Jul 31, 2018
e57d380
merge
gongweibao Jul 31, 2018
b77ee81
add
gongweibao Jul 31, 2018
397b453
fix
gongweibao Jul 31, 2018
c690f15
merge
gongweibao Sep 7, 2018
56a6a15
fix channel num
gongweibao Sep 7, 2018
6b638e0
fix inference compile
gongweibao Sep 7, 2018
0a4e750
fix anysis compile
gongweibao Sep 7, 2018
e9adf66
fix dist se
gongweibao Sep 7, 2018
839ef46
add checkpoint
gongweibao Sep 7, 2018
633718a
change initial conn num
gongweibao Sep 7, 2018
a4d489b
add sendcomplete
gongweibao Sep 28, 2018
dc07543
fix compile
gongweibao Sep 28, 2018
36df67d
fix
gongweibao Sep 29, 2018
0be5ca9
add
gongweibao Sep 29, 2018
949389f
Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…
gongweibao Sep 29, 2018
03476da
test=develop
gongweibao Sep 29, 2018
0154434
cleanup test=develop
gongweibao Sep 29, 2018
830079b
pre-commit tests=develop
gongweibao Sep 29, 2018
9afdf93
pre-commit test=develop
gongweibao Sep 29, 2018
cb087e4
Merge branch 'brpcserial' of https://github.com/gongweibao/Paddle int…
gongweibao Sep 29, 2018
a096876
fix gtest dependency test=develop
gongweibao Oct 8, 2018
8a5068a
Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…
gongweibao Oct 8, 2018
3309437
fix ci error test=develop
gongweibao Oct 8, 2018
fef1097
fix compile
gongweibao Oct 9, 2018
f5deaec
Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…
gongweibao Oct 9, 2018
c391d49
follow comments
gongweibao Oct 9, 2018
af74a19
add brpc
gongweibao Oct 11, 2018
cc46b3c
mrege
gongweibao Dec 6, 2018
e3ddd9a
fix ssl error
gongweibao Dec 6, 2018
92fb011
add some feature
gongweibao Dec 7, 2018
d260d5a
rm glob depends
gongweibao Dec 7, 2018
1fd988e
add record
gongweibao Dec 7, 2018
eecab31
fix trainer_id
gongweibao Dec 7, 2018
3585308
add checkpointnotify
gongweibao Dec 7, 2018
046c360
fix compile errors
gongweibao Dec 9, 2018
4a06e90
Merge branch 'develop' of https://github.com/PaddlePaddle/paddle into…
gongweibao Dec 11, 2018
eda4e36
add brpc collective server support
gongweibao Dec 11, 2018
2dd12aa
add collective server support
gongweibao Dec 11, 2018
f6bd695
fix grpc compile
gongweibao Dec 11, 2018
a76dc4e
fix cmake test=develop
gongweibao Dec 11, 2018
a7c7bf2
fix memory leak test=develop
gongweibao Dec 11, 2018
bc082c7
fix total limit warning test=develop
gongweibao Dec 11, 2018
13ca9c0
add more log
gongweibao Dec 12, 2018
7a4adad
split selected rows op should always init output selected rows
jacquesqiao Dec 10, 2018
a5534e1
test zero output of split_selected_rows_op
jacquesqiao Dec 11, 2018
c916534
fix the wrong format
tink2123 Nov 16, 2018
c559cd9
Fix the issue to run on AVX2 and AVX512F machines (#14851)
yihuaxu Dec 11, 2018
5a93241
Fix Eigen macro when using GPU
reyoung Dec 11, 2018
65c7d83
Fix compile error
reyoung Dec 11, 2018
f6b4015
Add gperftools supports for PE
reyoung Dec 10, 2018
db22ddc
Speed up SizeOfType
reyoung Dec 10, 2018
88acffc
Revert data_type
reyoung Dec 11, 2018
89394e2
Move ngraph compile control to cmake
baojun-nervana Dec 7, 2018
b5b1f90
Rename class
baojun-nervana Dec 7, 2018
d492699
Rename argument
baojun-nervana Dec 7, 2018
d6278a3
add async io
gongweibao Dec 13, 2018
e86f86b
clean up update mail fresh cla
gongweibao Dec 13, 2018
8c58ede
fresh test=develop
gongweibao Dec 14, 2018
f459d4a
fix
gongweibao Dec 14, 2018
842747a
follow comments test=develop
gongweibao Dec 14, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -153,8 +153,11 @@ include(external/cares)
if(WITH_DISTRIBUTE)
if(WITH_GRPC)
include(external/grpc)
message(STATUS "Use grpc framework.")
else()
message(STATUS "Use brpc framework.")
include(external/leveldb)
include(external/openssl)
include(external/brpc)
endif()
endif()
Expand Down
11 changes: 7 additions & 4 deletions cmake/external/brpc.cmake
Original file line number Diff line number Diff line change
Expand Up @@ -22,14 +22,16 @@ SET(BRPC_LIBRARIES "${BRPC_INSTALL_DIR}/lib/libbrpc.a" CACHE FILEPATH "brpc libr
INCLUDE_DIRECTORIES(${BRPC_INCLUDE_DIR})

# Reference https://stackoverflow.com/questions/45414507/pass-a-list-of-prefix-paths-to-externalproject-add-in-cmake-args
set(prefix_path "${THIRD_PARTY_PATH}/install/gflags|${THIRD_PARTY_PATH}/install/leveldb|${THIRD_PARTY_PATH}/install/snappy|${THIRD_PARTY_PATH}/install/gtest|${THIRD_PARTY_PATH}/install/protobuf")
set(prefix_path "${THIRD_PARTY_PATH}/install/gflags|${THIRD_PARTY_PATH}/install/leveldb|${THIRD_PARTY_PATH}/install/snappy|${THIRD_PARTY_PATH}/install/gtest|${THIRD_PARTY_PATH}/install/protobuf|${THIRD_PARTY_PATH}/install/openssl|${THIRD_PARTY_PATH}/install/zlib")

# If minimal .a is need, you can set WITH_DEBUG_SYMBOLS=OFF
ExternalProject_Add(
extern_brpc
${EXTERNAL_PROJECT_LOG_ARGS}
GIT_REPOSITORY "https://github.com/brpc/brpc"
GIT_TAG "6d153dd7ff00f960ae6895c9c5fff0ce9f07aff2"
#GIT_REPOSITORY "https://github.com/gongweibao/brpc"
#GIT_TAG "cd6d163cbce9260869b5f931df278fd137d47139"
GIT_REPOSITORY "https://github.com/Tuvie/brpc"
GIT_TAG "5385f8c3da55b78c3d21873601b7662d3033aaf7"
PREFIX ${BRPC_SOURCES_DIR}
UPDATE_COMMAND ""
CMAKE_ARGS -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
Expand All @@ -42,14 +44,15 @@ ExternalProject_Add(
-DCMAKE_BUILD_TYPE=${THIRD_PARTY_BUILD_TYPE}
-DCMAKE_PREFIX_PATH=${prefix_path}
-DBRPC_WITH_GLOG=ON
-DIOBUF_WITH_HUGE_BLOCK=ON
${EXTERNAL_OPTIONAL_ARGS}
LIST_SEPARATOR |
CMAKE_CACHE_ARGS -DCMAKE_INSTALL_PREFIX:PATH=${BRPC_INSTALL_DIR}
-DCMAKE_INSTALL_LIBDIR:PATH=${BRPC_INSTALL_DIR}/lib
-DCMAKE_POSITION_INDEPENDENT_CODE:BOOL=ON
-DCMAKE_BUILD_TYPE:STRING=${THIRD_PARTY_BUILD_TYPE}
)
ADD_DEPENDENCIES(extern_brpc protobuf leveldb gflags glog gtest snappy)
ADD_DEPENDENCIES(extern_brpc protobuf leveldb gflags glog gtest snappy ssl crypto)
ADD_LIBRARY(brpc STATIC IMPORTED GLOBAL)
SET_PROPERTY(TARGET brpc PROPERTY IMPORTED_LOCATION ${BRPC_LIBRARIES})
ADD_DEPENDENCIES(brpc extern_brpc)
Expand Down
50 changes: 50 additions & 0 deletions cmake/external/openssl.cmake
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# Copyright (c) 2016 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

IF(MOBILE_INFERENCE OR NOT WITH_DISTRIBUTE)
return()
ENDIF()

include (ExternalProject)

SET(OPENSSL_SOURCES_DIR ${THIRD_PARTY_PATH}/openssl)
SET(OPENSSL_INSTALL_DIR ${THIRD_PARTY_PATH}/install/openssl)
SET(OPENSSL_INCLUDE_DIR "${OPENSSL_INSTALL_DIR}/include/" CACHE PATH "openssl include directory." FORCE)
SET(OPENSSL_LIBRARIES "${OPENSSL_INSTALL_DIR}/lib/libssl.a" CACHE FILEPATH "ssl library." FORCE)
SET(CRYPTO_LIBRARIES "${OPENSSL_INSTALL_DIR}/lib/libcrypto.a" CACHE FILEPATH "ssl library." FORCE)

ExternalProject_Add(
extern_openssl
${EXTERNAL_PROJECT_LOG_ARGS}
GIT_REPOSITORY "https://github.com/openssl/openssl"
GIT_TAG "c35608e5422d2718868d88439e22369d4aabb7c6"
PREFIX ${OPENSSL_SOURCES_DIR}
UPDATE_COMMAND ""
INSTALL_DIR ${OPENSSL_INSTALL_DIR}
CONFIGURE_COMMAND ./config -fPIC no-shared --prefix=${OPENSSL_INSTALL_DIR}
BUILD_IN_SOURCE 1
BUILD_COMMAND $(MAKE) -j 12
INSTALL_COMMAND $(MAKE) install
)

ADD_DEPENDENCIES(extern_openssl zlib)
ADD_LIBRARY(ssl STATIC IMPORTED GLOBAL)
ADD_LIBRARY(crypto STATIC IMPORTED GLOBAL)
SET_PROPERTY(TARGET ssl PROPERTY IMPORTED_LOCATION ${OPENSSL_LIBRARIES})
SET_PROPERTY(TARGET crypto PROPERTY IMPORTED_LOCATION ${CRYPTO_LIBRARIES})
ADD_DEPENDENCIES(ssl extern_ssl)


LIST(APPEND external_project_dependencies ssl)
7 changes: 3 additions & 4 deletions paddle/contrib/inference/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,7 @@ if(APPLE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-error=pessimizing-move")
endif(APPLE)


set(inference_deps paddle_inference_api paddle_fluid_api)
set(inference_deps paddle_inference_api paddle_fluid)

function(inference_api_test TARGET_NAME)
if (WITH_TESTING)
Expand All @@ -30,7 +29,7 @@ function(inference_api_test TARGET_NAME)
set(PYTHON_TESTS_DIR ${PADDLE_BINARY_DIR}/python/paddle/fluid/tests)
cc_test(${TARGET_NAME}
SRCS ${TARGET_NAME}.cc
DEPS "${inference_deps}"
DEPS "${inference_deps}" ${GLOB_DISTRIBUTE_DEPS}
ARGS --dirname=${PYTHON_TESTS_DIR}/book/)
if(inference_test_ARGS)
set_tests_properties(${TARGET_NAME}
Expand All @@ -45,7 +44,7 @@ cc_library(paddle_inference_api

cc_test(test_paddle_inference_api
SRCS test_paddle_inference_api.cc
DEPS paddle_inference_api)
DEPS paddle_inference_api paddle_fluid ${GLOB_DISTRIBUTE_DEPS})

inference_api_test(test_paddle_inference_api_impl
ARGS test_word2vec test_image_classification)
Expand Down
8 changes: 6 additions & 2 deletions paddle/fluid/framework/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,11 @@ cc_library(lod_rank_table SRCS lod_rank_table.cc DEPS lod_tensor)
cc_library(feed_fetch_method SRCS feed_fetch_method.cc DEPS lod_tensor scope glog)

if(WITH_DISTRIBUTE)
cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto glog lod_rank_table feed_fetch_method sendrecvop_grpc cares grpc++_unsecure grpc_unsecure gpr)
if(WITH_GRPC)
cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto glog lod_rank_table feed_fetch_method ${GLOB_DISTRIBUTE_DEPS})
else(WITH_GRPC)
cc_library(executor SRCS executor.cc DEPS op_registry device_context scope framework_proto glog lod_rank_table feed_fetch_method ${GLOB_DISTRIBUTE_DEPS})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to be different?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Done!

endif()
set(DISTRIBUTE_COMPILE_FLAGS "-Wno-non-virtual-dtor -Wno-error=non-virtual-dtor -Wno-error=delete-non-virtual-dtor")
set_source_files_properties(executor.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})
else()
Expand All @@ -111,4 +115,4 @@ cc_test(cow_ptr_tests SRCS details/cow_ptr_test.cc)
cc_test(tuple_test SRCS tuple_test.cc )
cc_test(concurrency_test SRCS concurrency_test.cc DEPS go_op channel_close_op channel_create_op
channel_send_op channel_recv_op sum_op select_op elementwise_add_op compare_op
conditional_block_op while_op assign_op print_op executor proto_desc)
conditional_block_op while_op assign_op print_op executor proto_desc ${GLOB_DISTRIBUTE_DEPS})
7 changes: 2 additions & 5 deletions paddle/fluid/framework/executor.cc
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,7 @@ limitations under the License. */
#include "paddle/fluid/framework/lod_tensor_array.h"
#include "paddle/fluid/framework/op_registry.h"
#include "paddle/fluid/framework/reader.h"
#ifdef PADDLE_WITH_DISTRIBUTE
#include "paddle/fluid/operators/distributed/grpc_client.h"
#endif
#include "paddle/fluid/operators/detail/macros.h"
#include "paddle/fluid/platform/place.h"
#include "paddle/fluid/platform/profiler.h"

Expand All @@ -49,8 +47,7 @@ Executor::Executor(const platform::Place& place) : place_(place) {}

#ifdef PADDLE_WITH_DISTRIBUTE
void Executor::Complete() {
::paddle::operators::distributed::RPCClient::GetInstance<
::paddle::operators::distributed::GRPCClient>()
::paddle::operators::distributed::RPCClient::GetInstance<RPCCLIENT_T>()
->SendComplete();
}
#endif
Expand Down
2 changes: 1 addition & 1 deletion paddle/fluid/framework/lod_tensor.cc
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ std::ostream &operator<<(std::ostream &os, const LoDTensor &t) {
// only print first ten elements
int64_t size = t.numel() < 10 ? t.numel() : 10;
for (int64_t i = 0; i < size; ++i) {
if (t.type().hash_code() == typeid(float).hash_code()) {
if (t.type().hash_code() == typeid(float).hash_code()) { // NOLINT
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can put these fix in another PR to reduce this PR's size?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ye.Will do it!

os << t.data<float>()[i] << " ";
} else if (t.type().hash_code() == typeid(int64_t).hash_code()) {
os << t.data<int64_t>()[i] << " ";
Expand Down
3 changes: 2 additions & 1 deletion paddle/fluid/inference/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
set(FLUID_CORE_MODULES proto_desc memory lod_tensor executor init)

# TODO(panyx0718): Should this be called paddle_fluid_inference_api_internal?
# If WITH_DISTRIBUTE=OFF, DISTRIBUTE_DEPS is "". So don't worry about it.
cc_library(paddle_fluid_api
SRCS io.cc
DEPS ${FLUID_CORE_MODULES} ${GLOB_OP_LIB})
Expand All @@ -12,7 +13,7 @@ if(WITH_CONTRIB)
endif()

# Create static library
cc_library(paddle_fluid DEPS ${fluid_modules} paddle_fluid_api)
cc_library(paddle_fluid DEPS ${fluid_modules} paddle_fluid_api ${GLOB_DISTRIBUTE_DEPS})
# Create shared library
cc_library(paddle_fluid_shared SHARED
SRCS io.cc
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@

#pragma once

#include <string>
#include "paddle/fluid/framework/program_desc.h"
#include "paddle/fluid/inference/analysis/data_flow_graph.h"
#include "paddle/fluid/inference/analysis/pass.h"
Expand Down
2 changes: 1 addition & 1 deletion paddle/fluid/inference/analysis/dfg_graphviz_draw_pass.h
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ class DFG_GraphvizDrawPass : public DataFlowGraphPass {
const bool display_deleted_node;
};

DFG_GraphvizDrawPass(const Config &config) : config_(config) {}
explicit DFG_GraphvizDrawPass(const Config &config) : config_(config) {}

bool Initialize(Argument *argument) override { return true; }
void Run(DataFlowGraph *graph) override;
Expand Down
6 changes: 3 additions & 3 deletions paddle/fluid/inference/analysis/pass_manager_tester.cc
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,14 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. */

#include "paddle/fluid/inference/analysis/pass_manager.h"
#include <gtest/gtest.h>

#include "paddle/fluid/inference/analysis/data_flow_graph_to_fluid_pass.h"
#include "paddle/fluid/inference/analysis/dfg_graphviz_draw_pass.h"
#include "paddle/fluid/inference/analysis/fluid_to_data_flow_graph_pass.h"
#include "paddle/fluid/inference/analysis/pass_manager.h"
#include "paddle/fluid/inference/analysis/ut_helper.h"

#include <gtest/gtest.h>

namespace paddle {
namespace inference {
namespace analysis {
Expand Down
4 changes: 2 additions & 2 deletions paddle/fluid/inference/analysis/tensorrt_subgraph_pass.h
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ class TensorRTSubGraphPass : public DataFlowGraphPass {
// Tell whether to transform a sub-graph into TensorRT.
using NodeInsideSubgraphTeller = SubGraphFuse::NodeInsideSubgraphTeller;

TensorRTSubGraphPass(const NodeInsideSubgraphTeller& teller);
explicit TensorRTSubGraphPass(const NodeInsideSubgraphTeller& teller);

bool Initialize(Argument* argument) override { return true; }

Expand All @@ -44,4 +44,4 @@ class TensorRTSubGraphPass : public DataFlowGraphPass {

} // namespace analysis
} // namespace inference
} // paddle
} // namespace paddle
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ TEST(TensorRTSubGraph, single_pass) {

// Check the TRT op's block desc
for (auto node : dfg.nodes.nodes()) {
if (node->IsFunctionBlock()) {
if (node->IsFunctionBlock()) { // NOLINT
}
}
}
Expand Down
4 changes: 2 additions & 2 deletions paddle/fluid/inference/tests/book/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ function(inference_test TARGET_NAME)
string(REGEX REPLACE "^_$" "" arg "${arg}")
cc_test(test_inference_${TARGET_NAME}${arg}
SRCS test_inference_${TARGET_NAME}.cc
DEPS paddle_fluid
DEPS paddle_fluid ${GLOB_DISTRIBUTE_DEPS}
ARGS --dirname=${PYTHON_TESTS_DIR}/book/${TARGET_NAME}${arg}.inference.model)
set_tests_properties(test_inference_${TARGET_NAME}${arg}
PROPERTIES DEPENDS test_${TARGET_NAME})
Expand All @@ -43,6 +43,6 @@ inference_test(word2vec)
# TODO(TJ): clean me up
cc_test(test_inference_nlp
SRCS test_inference_nlp.cc
DEPS paddle_fluid
DEPS paddle_fluid ${GLOB_DISTRIBUTE_DEPS}
ARGS
--model_path=${PADDLE_BINARY_DIR}/python/paddle/fluid/tests/book/recognize_digits_mlp.inference.model)
4 changes: 3 additions & 1 deletion paddle/fluid/operators/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,7 @@ else()
set(DEPS_OPS ${DEPS_OPS} nccl_op)
endif()

set(DISTRIBUTE_DEPS "")
if(WITH_DISTRIBUTE)
add_subdirectory(distributed)

Expand All @@ -205,7 +206,7 @@ if(WITH_DISTRIBUTE)
# listen_and_serv_op sum_op executor SERIAL)
if(WITH_GPU)
set_source_files_properties(test_send_nccl_id.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})
cc_test(test_send_nccl_id SRCS test_send_nccl_id.cc DEPS listen_and_serv_op executor SERIAL)
cc_test(test_send_nccl_id SRCS test_send_nccl_id.cc DEPS listen_and_serv_op ${DISTRIBUTE_DEPS} executor SERIAL)
if(WITH_GRPC)
op_library(gen_nccl_id_op DEPS nccl_common sendrecvop_grpc)
else()
Expand Down Expand Up @@ -296,6 +297,7 @@ foreach(src ${DETECTION_LIBRARY})
endforeach()

set(GLOB_OP_LIB ${OP_LIBRARY} CACHE INTERNAL "Global OP library")
set(GLOB_DISTRIBUTE_DEPS ${DISTRIBUTE_DEPS} CACHE INTERNAL "distributed dependency")

cc_test(gather_test SRCS gather_test.cc DEPS tensor)
cc_test(scatter_test SRCS scatter_test.cc DEPS tensor)
Expand Down
20 changes: 14 additions & 6 deletions paddle/fluid/operators/detail/macros.h
Original file line number Diff line number Diff line change
Expand Up @@ -14,14 +14,22 @@

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

macros.h文件是只给rpc相关使用,还是其他OP也能使用, 如果其他OP也能用,不需要RPC相关的OP引入RPC相关的头文件感觉不合适

#pragma once

#ifdef PADDLE_WITH_DISTRIBUTE

#ifdef PADDLE_WITH_GRPC

#include "paddle/fluid/operators/distributed/grpc_client.h"
#include "paddle/fluid/operators/distributed/grpc_server.h"
#define RPCSERVER_T distributed::AsyncGRPCServer
#define RPCCLIENT_T distributed::GRPCClient
#else
#define RPCSERVER_T paddle::operators::distributed::AsyncGRPCServer
#define RPCCLIENT_T paddle::operators::distributed::GRPCClient

#else // PADDLE_WITH_GRPC

#include "paddle/fluid/operators/distributed/brpc_client.h"
#include "paddle/fluid/operators/distributed/brpc_server.h"
#define RPCSERVER_T distributed::AsyncBRPCServer
#define RPCCLIENT_T distributed::BRPCClient
#endif
#define RPCSERVER_T paddle::operators::distributed::AsyncBRPCServer
#define RPCCLIENT_T paddle::operators::distributed::BRPCClient

#endif // PADDLE_WITH_GRPC

#endif // PADDLE_WITH_DISTRIBUTE
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ limitations under the License. */
syntax = "proto3";
package sendrecv;

// option cc_generic_services = true;
option cc_generic_services = true;

service SendRecvService {
// For parameter server round-robin like hashing, do not split tensors.
Expand Down
49 changes: 29 additions & 20 deletions paddle/fluid/operators/distributed/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -1,33 +1,42 @@
if(NOT WITH_DISTRIBUTE)
return()
endif()

if(WITH_GRPC)
grpc_library(sendrecvop_grpc SRCS bytebuffer_stream.cc sendrecvop_utils.cc grpc_client.cc
request_handler_impl.cc rpc_client.cc rpc_server.cc grpc_server.cc variable_response.cc PROTO send_recv.proto DEPS lod_tensor
selected_rows memory)
set(cc_generic_services "false")
else()
set(cc_generic_services "true")
endif()
configure_file(send_recv.proto.in ${CMAKE_CURRENT_SOURCE_DIR}/send_recv.proto @ONLY)

if(WITH_GRPC)
grpc_library(sendrecvop_grpc SRCS grpc_bytebuffer_stream.cc sendrecvop_utils.cc grpc_client.cc
request_handler_impl.cc rpc_client.cc rpc_server.cc grpc_server.cc variable_response.cc grpc_variable_response.cc grpc_sendrecvop_utils.cc
PROTO send_recv.proto
DEPS lod_tensor selected_rows memory)
set(DISTRIBUTE_COMPILE_FLAGS "-Wno-non-virtual-dtor -Wno-error=non-virtual-dtor -Wno-error=delete-non-virtual-dtor")
set_source_files_properties(grpc_serde_test.cc rpc_server_test.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})
cc_test(serde_test SRCS grpc_serde_test.cc variable_response.cc DEPS grpc++_unsecure grpc_unsecure gpr
cares zlib protobuf sendrecvop_grpc SERIAL)
cc_test(grpc_server_test SRCS rpc_server_test.cc DEPS sendrecvop_grpc
grpc++_unsecure grpc_unsecure gpr cares zlib protobuf executor
proto_desc lookup_table_op SERIAL)
cc_test(grpc_serde_test SRCS grpc_serde_test.cc
DEPS grpc++_unsecure grpc_unsecure gpr cares zlib protobuf sendrecvop_grpc SERIAL)
cc_test(grpc_server_test SRCS rpc_server_test.cc
DEPS sendrecvop_grpc grpc++_unsecure grpc_unsecure gpr cares zlib protobuf executor proto_desc lookup_table_op SERIAL)
return()
endif()


set(DISTRIBUTE_COMPILE_FLAGS "-Wno-non-virtual-dtor -Wno-error=non-virtual-dtor -Wno-error=delete-non-virtual-dtor")
set_source_files_properties(brpc_server.cc brpc_client.cc rpc_server_test.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})
brpc_library(sendrecvop_brpc SRCS brpc_client.cc brpc_server.cc rpc_server.cc rpc_client.cc request_handler_impl.cc
set_source_files_properties(brpc_server.cc brpc_client.cc rpc_server_test.cc brpc_serde_test.cc
brpc_variable_response.cc brpc_sendrecvop_utils.cc PROPERTIES COMPILE_FLAGS ${DISTRIBUTE_COMPILE_FLAGS})

brpc_library(sendrecvop_brpc SRCS brpc_client.cc brpc_server.cc rpc_server.cc rpc_client.cc request_handler_impl.cc brpc_sendrecvop_utils.cc
brpc_variable_response.cc variable_response.cc sendrecvop_utils.cc
PROTO send_recv.proto
DEPS lod_tensor selected_rows memory)

find_library(OPENSSL_CRYPTO_LIBRARY_STATIC NAMES libcrypto.so)
ADD_LIBRARY(crypto SHARED IMPORTED GLOBAL)
SET_PROPERTY(TARGET crypto PROPERTY IMPORTED_LOCATION ${OPENSSL_CRYPTO_LIBRARY_STATIC})

set(brpc_test_depends sendrecvop_brpc brpc protobuf leveldb gflags glog executor proto_desc lookup_table_op snappystream snappy ssl crypto)

find_library(OPENSSL_SSL_LIBRARY_STATIC NAMES libssl.so)
ADD_LIBRARY(ssl SHARED IMPORTED GLOBAL)
SET_PROPERTY(TARGET ssl PROPERTY IMPORTED_LOCATION ${OPENSSL_SSL_LIBRARY_STATIC})
cc_test(brpc_server_test SRCS rpc_server_test.cc
DEPS ${brpc_test_depends} SERIAL)

cc_test(brpc_server_test SRCS rpc_server_test.cc DEPS sendrecvop_brpc
brpc protobuf leveldb gflags glog
protobuf executor proto_desc lookup_table_op snappystream snappy ssl crypto SERIAL)
cc_test(brpc_serde_test SRCS brpc_serde_test.cc
DEPS ${brpc_test_depends} SERIAL)
Loading