Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

distribute trainer's rpc client cannot wait server to start up #11568

Closed
xuezhong opened this issue Jun 19, 2018 · 1 comment
Closed

distribute trainer's rpc client cannot wait server to start up #11568

xuezhong opened this issue Jun 19, 2018 · 1 comment

Comments

@xuezhong
Copy link
Contributor

rpc client should try to establish connection in period while rpc server is not started. Following is the log:
init fluid.framework.default_startup_program
E0619 09:11:08.576251 159 grpc_client.cc:242] proc param error:name:[fc_75.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576308 192 grpc_client.cc:242] proc param error:name:[fc_64.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576380 194 grpc_client.cc:242] proc param error:name:[fc_19.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576922 192 grpc_client.cc:242] proc param error:name:[fc_54.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576419 134 grpc_client.cc:242] proc param error:name:[fc_7.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576463 196 grpc_client.cc:242] proc param error:name:[fc_4.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577029 147 grpc_client.cc:242] proc param error:name:[fc_61.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576557 122 grpc_client.cc:242] proc param error:name:[layer_norm_22.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576613 145 grpc_client.cc:242] proc param error:name:[layer_norm_22.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577158 156 grpc_client.cc:242] proc param error:name:[fc_51.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576670 139 grpc_client.cc:242] proc param error:name:[fc_55.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576730 162 grpc_client.cc:242] proc param error:name:[fc_60.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576750 212 grpc_client.cc:242] proc param error:name:[layer_norm_21.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576810 167 grpc_client.cc:242] proc param error:name:[fc_35.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576843 203 grpc_client.cc:242] proc param error:name:[fc_58.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576901 120 grpc_client.cc:242] proc param error:name:[layer_norm_20.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576381 208 grpc_client.cc:242] proc param error:name:[fc_64.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576982 173 grpc_client.cc:242] proc param error:name:[fc_59.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577392 208 grpc_client.cc:242] proc param error:name:[fc_35.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577075 165 grpc_client.cc:242] proc param error:name:[fc_6.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576647 202 grpc_client.cc:242] proc param error:name:[fc_3.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577174 184 grpc_client.cc:242] proc param error:name:[layer_norm_27.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577208 156 grpc_client.cc:242] proc param error:name:[fc_52.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577270 181 grpc_client.cc:242] proc param error:name:[fc_48.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577338 182 grpc_client.cc:242] proc param error:name:[fc_56.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577540 180 grpc_client.cc:242] proc param error:name:[fc_42.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.576500 115 grpc_client.cc:242] proc param error:name:[fc_63.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577435 131 grpc_client.cc:242] proc param error:name:[fc_50.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577486 202 grpc_client.cc:242] proc param error:name:[fc_23.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577881 131 grpc_client.cc:242] proc param error:name:[fc_79.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577618 142 grpc_client.cc:242] proc param error:name:[fc_41.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578013 114 grpc_client.cc:242] proc param error:name:[fc_74.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578068 148 grpc_client.cc:242] proc param error:name:[layer_norm_14.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577742 159 grpc_client.cc:242] proc param error:name:[fc_28.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577772 201 grpc_client.cc:242] proc param error:name:[fc_40.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577801 194 grpc_client.cc:242] proc param error:name:[fc_44.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577860 192 grpc_client.cc:242] proc param error:name:[fc_43.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577352 155 grpc_client.cc:242] proc param error:name:[layer_norm_18.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577940 130 grpc_client.cc:242] proc param error:name:[fc_68.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577980 199 grpc_client.cc:242] proc param error:name:[fc_77.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577642 143 grpc_client.cc:242] proc param error:name:[fc_5.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578431 210 grpc_client.cc:242] proc param error:name:[fc_31.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578089 114 grpc_client.cc:242] proc param error:name:[layer_norm_11.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578141 207 grpc_client.cc:242] proc param error:name:[fc_75.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578197 197 grpc_client.cc:242] proc param error:name:[fc_72.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578557 212 grpc_client.cc:242] proc param error:name:[fc_34.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578284 145 grpc_client.cc:242] proc param error:name:[fc_37.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578630 203 grpc_client.cc:242] proc param error:name:[fc_30.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578374 137 grpc_client.cc:242] proc param error:name:[fc_73.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578397 153 grpc_client.cc:242] proc param error:name:[fc_22.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.577673 191 grpc_client.cc:242] proc param error:name:[layer_norm_16.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578469 190 grpc_client.cc:242] proc param error:name:[fc_54.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578508 162 grpc_client.cc:242] proc param error:name:[fc_29.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578236 179 grpc_client.cc:242] proc param error:name:[fc_65.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578909 177 grpc_client.cc:242] proc param error:name:[fc_11.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578325 206 grpc_client.cc:242] proc param error:name:[fc_67.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578675 204 grpc_client.cc:242] proc param error:name:[fc_32.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578727 152 grpc_client.cc:242] proc param error:name:[fc_44.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578759 117 grpc_client.cc:242] proc param error:name:[fc_46.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578799 169 grpc_client.cc:242] proc param error:name:[fc_14.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578812 191 grpc_client.cc:242] proc param error:name:[layer_norm_0.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578876 135 grpc_client.cc:242] proc param error:name:[fc_49.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578573 197 grpc_client.cc:242] proc param error:name:[fc_1.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578956 157 grpc_client.cc:242] proc param error:name:[fc_27.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.578994 150 grpc_client.cc:242] proc param error:name:[fc_12.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579047 187 grpc_client.cc:242] proc param error:name:[fc_23.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579103 189 grpc_client.cc:242] proc param error:name:[fc_20.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579149 186 grpc_client.cc:242] proc param error:name:[fc_29.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579179 129 grpc_client.cc:242] proc param error:name:[fc_18.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579217 185 grpc_client.cc:242] proc param error:name:[fc_33.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579254 128 grpc_client.cc:242] proc param error:name:[embedding_0.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579293 144 grpc_client.cc:242] proc param error:name:[fc_34.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579342 135 grpc_client.cc:242] proc param error:name:[fc_21.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579368 180 grpc_client.cc:242] proc param error:name:[fc_10.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579404 132 grpc_client.cc:242] proc param error:name:[fc_25.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579460 146 grpc_client.cc:242] proc param error:name:[fc_17.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579489 141 grpc_client.cc:242] proc param error:name:[fc_16.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579536 170 grpc_client.cc:242] proc param error:name:[fc_16.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579563 189 grpc_client.cc:242] proc param error:name:[embedding_2.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579622 121 grpc_client.cc:242] proc param error:name:[fc_13.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579670 116 grpc_client.cc:242] proc param error:name:[fc_24.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579713 149 grpc_client.cc:242] proc param error:name:[fc_26.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579758 118 grpc_client.cc:242] proc param error:name:[fc_0.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579792 195 grpc_client.cc:242] proc param error:name:[fc_11.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579838 147 grpc_client.cc:242] proc param error:name:[fc_15.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579875 140 grpc_client.cc:242] proc param error:name:[fc_28.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579921 211 grpc_client.cc:242] proc param error:name:[fc_22.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.579954 122 grpc_client.cc:242] proc param error:name:[fc_38.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:08.580003 174 grpc_client.cc:242] proc param error:name:[fc_10.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539103 125 grpc_client.cc:242] proc param error:name:[layer_norm_9.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539139 198 grpc_client.cc:242] proc param error:name:[layer_norm_1.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539183 155 grpc_client.cc:242] proc param error:name:[layer_norm_8.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539216 130 grpc_client.cc:242] proc param error:name:[layer_norm_7.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539260 199 grpc_client.cc:242] proc param error:name:[fc_92.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539335 143 grpc_client.cc:242] proc param error:name:[layer_norm_5.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539346 210 grpc_client.cc:242] proc param error:name:[layer_norm_29.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539382 114 grpc_client.cc:242] proc param error:name:[fc_83.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539399 205 grpc_client.cc:242] proc param error:name:[layer_norm_28.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539458 167 grpc_client.cc:242] proc param error:name:[layer_norm_26.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539490 212 grpc_client.cc:242] proc param error:name:[layer_norm_19.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539542 145 grpc_client.cc:242] proc param error:name:[layer_norm_25.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539598 203 grpc_client.cc:242] proc param error:name:[layer_norm_7.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539677 137 grpc_client.cc:242] proc param error:name:[fc_74.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539738 153 grpc_client.cc:242] proc param error:name:[layer_norm_21.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539798 173 grpc_client.cc:242] proc param error:name:[layer_norm_20.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539862 190 grpc_client.cc:242] proc param error:name:[layer_norm_27.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539901 162 grpc_client.cc:242] proc param error:name:[fc_53.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.539934 179 grpc_client.cc:242] proc param error:name:[layer_norm_25.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540012 177 grpc_client.cc:242] proc param error:name:[layer_norm_19.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540066 184 grpc_client.cc:242] proc param error:name:[layer_norm_17.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540133 206 grpc_client.cc:242] proc param error:name:[fc_5.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540204 156 grpc_client.cc:242] proc param error:name:[layer_norm_17.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540237 152 grpc_client.cc:242] proc param error:name:[layer_norm_2.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540271 117 grpc_client.cc:242] proc param error:name:[layer_norm_13.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540308 169 grpc_client.cc:242] proc param error:name:[layer_norm_12.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540364 191 grpc_client.cc:242] proc param error:name:[layer_norm_14.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540393 197 grpc_client.cc:242] proc param error:name:[layer_norm_13.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540446 157 grpc_client.cc:242] proc param error:name:[fc_17.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540484 136 grpc_client.cc:242] proc param error:name:[layer_norm_8.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540513 187 grpc_client.cc:242] proc param error:name:[fc_96.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540561 127 grpc_client.cc:242] proc param error:name:[fc_95.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540586 186 grpc_client.cc:242] proc param error:name:[fc_94.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540627 129 grpc_client.cc:242] proc param error:name:[fc_93.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540637 151 grpc_client.cc:242] proc param error:name:[fc_90.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540704 128 grpc_client.cc:242] proc param error:name:[fc_88.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540735 144 grpc_client.cc:242] proc param error:name:[fc_87.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540781 135 grpc_client.cc:242] proc param error:name:[fc_85.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540812 180 grpc_client.cc:242] proc param error:name:[fc_84.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540849 132 grpc_client.cc:242] proc param error:name:[fc_84.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540884 146 grpc_client.cc:242] proc param error:name:[fc_81.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540915 159 grpc_client.cc:242] proc param error:name:[fc_80.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540964 189 grpc_client.cc:242] proc param error:name:[fc_45.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.540989 116 grpc_client.cc:242] proc param error:name:[layer_norm_28.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541033 118 grpc_client.cc:242] proc param error:name:[fc_4.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541064 126 grpc_client.cc:242] proc param error:name:[fc_71.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541100 194 grpc_client.cc:242] proc param error:name:[fc_70.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541146 160 grpc_client.cc:242] proc param error:name:[fc_47.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541187 183 grpc_client.cc:242] proc param error:name:[fc_66.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541229 164 grpc_client.cc:242] proc param error:name:[fc_39.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541257 207 grpc_client.cc:242] proc param error:name:[layer_norm_24.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541287 172 grpc_client.cc:242] proc param error:name:[fc_76.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541328 120 grpc_client.cc:242] proc param error:name:[fc_45.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541354 168 grpc_client.cc:242] proc param error:name:[fc_8.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541401 208 grpc_client.cc:242] proc param error:name:[fc_78.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541447 165 grpc_client.cc:242] proc param error:name:[layer_norm_2.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541473 176 grpc_client.cc:242] proc param error:name:[fc_65.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541512 204 grpc_client.cc:242] proc param error:name:[fc_82.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541570 133 grpc_client.cc:242] proc param error:name:[layer_norm_29.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541613 188 grpc_client.cc:242] proc param error:name:[fc_85.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541651 166 grpc_client.cc:242] proc param error:name:[fc_86.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541683 119 grpc_client.cc:242] proc param error:name:[fc_89.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541718 138 grpc_client.cc:242] proc param error:name:[layer_norm_5.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541743 202 grpc_client.cc:242] proc param error:name:[layer_norm_6.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541798 196 grpc_client.cc:242] proc param error:name:[fc_91.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541833 134 grpc_client.cc:242] proc param error:name:[layer_norm_6.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541868 141 grpc_client.cc:242] proc param error:name:[fc_94.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541913 121 grpc_client.cc:242] proc param error:name:[fc_95.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541939 200 grpc_client.cc:242] proc param error:name:[layer_norm_0.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.541981 192 grpc_client.cc:242] proc param error:name:[layer_norm_9.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542013 139 grpc_client.cc:242] proc param error:name:[layer_norm_1.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542055 171 grpc_client.cc:242] proc param error:name:[layer_norm_11.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542099 175 grpc_client.cc:242] proc param error:name:[layer_norm_10.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542131 113 grpc_client.cc:242] proc param error:name:[layer_norm_10.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542167 209 grpc_client.cc:242] proc param error:name:[fc_69.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542208 182 grpc_client.cc:242] proc param error:name:[layer_norm_18.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542243 150 grpc_client.cc:242] proc param error:name:[layer_norm_12.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542282 123 grpc_client.cc:242] proc param error:name:[layer_norm_15.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542316 131 grpc_client.cc:242] proc param error:name:[layer_norm_15.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542366 148 grpc_client.cc:242] proc param error:name:[fc_57.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542409 149 grpc_client.cc:242] proc param error:name:[fc_62.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542428 178 grpc_client.cc:242] proc param error:name:[fc_55.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542470 163 grpc_client.cc:242] proc param error:name:[layer_norm_24.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542512 124 grpc_client.cc:242] proc param error:name:[layer_norm_23.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542547 115 grpc_client.cc:242] proc param error:name:[fc_36.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542593 185 grpc_client.cc:242] proc param error:name:[layer_norm_26.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542618 170 grpc_client.cc:242] proc param error:name:[fc_2.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542660 158 grpc_client.cc:242] proc param error:name:[layer_norm_16.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542685 181 grpc_client.cc:242] proc param error:name:[layer_norm_3.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542732 142 grpc_client.cc:242] proc param error:name:[layer_norm_3.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542765 154 grpc_client.cc:242] proc param error:name:[layer_norm_4.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542794 201 grpc_client.cc:242] proc param error:name:[layer_norm_4.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542847 161 grpc_client.cc:242] proc param error:name:[fc_9.w_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
E0619 09:11:09.542887 193 grpc_client.cc:242] proc param error:name:[layer_norm_23.b_0@GRAD.trainer_1] ep:[127.0.0.1:6176] grpc error:Connect Failed
Traceback (most recent call last):
File "thirdparty/model/transformer_cloud/train.py", line 493, in
train(args)
File "thirdparty/model/transformer_cloud/train.py", line 486, in train
train_loop(exe, trainer_prog)
File "thirdparty/model/transformer_cloud/train.py", line 420, in train_loop
outs = train_exe.run(fetch_list=[sum_cost.name, token_num.name], feed=feed_list)
File "/usr/local/lib/python2.7/dist-packages/paddle/fluid/parallel_executor.py", line 238, in run
self.executor.run(fetch_list, fetch_var_name)
paddle.fluid.core.EnforceNotMet: at [/paddle/Paddle/paddle/fluid/operators/send_op.cc:67]
PaddlePaddle Call Stacks:
0 0x7f563fc4e4dcp paddle::platform::EnforceNotMet::EnforceNotMet(std::exception_ptr::exception_ptr, char const*, int) + 572
1 0x7f5640a0386fp paddle::operators::SendOp::RunImpl(paddle::framework::Scope const&, boost::variant<paddle::platform::CUDAPlace, paddle::platform::CPUPlace, paddle::platform::CUDAPinnedPlace, boost::detail::variant::void, boost::detail::variant::void, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_, boost::detail::variant::void_> const&) const + 5359
2 0x7f5640a1cf80p paddle::framework::details::SendOpHandle::RunImpl() + 1376
3 0x7f5640a41f61p paddle::framework::details::OpHandleBase::Run(bool) + 321
4 0x7f5640a3713cp
5 0x7f5640a37690p
6 0x7f564089fadep std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) + 46
7 0x7f567ae4ba99p
8 0x7f5640a3640dp
9 0x7f5640a3c4d4p std::thread::_Impl<std::_Bind_simple<ThreadPool::ThreadPool(unsigned long)::{lambda()#1} ()> >::_M_run() + 340
10 0x7f567347fc80p
11 0x7f567ae446bap
12 0x7f567ab7a41dp clone + 109
@shanyi15
Copy link
Collaborator

您好，此issue在近一个月内暂无更新，我们将于今天内关闭。若在关闭后您仍需跟进提问，可重新开启此问题，我们将在24小时内回复您。因关闭带来的不便我们深表歉意，请您谅解~感谢您对PaddlePaddle的支持!
Hello, this issue has not been updated in the past month. We will close it today for the sake of other user‘s experience. If you still need to follow up on this question after closing, please feel free to reopen it. In that case, we will get back to you within 24 hours. We apologize for the inconvenience caused by the closure and thank you so much for your support of PaddlePaddle Group!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@xuezhong @shanyi15