Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

saga事务进入无限补偿重试,1个补偿大致每秒重试80次,请问是什么原因? #306

Closed
mowocc opened this issue Jun 13, 2022 · 15 comments

Comments

@mowocc
Copy link

mowocc commented Jun 13, 2022

问题:saga事务进入无限补偿重试,1个补偿大致每秒重试80次,请问是什么原因?
事务:

  • /goods.service.v1.Goods/StockDeduct商品库存扣减,/goods.service.v1.Goods/StockDeductRevert商品库存扣减补偿
  • /order.service.v1.Order/Create订单创建,/order.service.v1.Order/CreateRevert订单创建补偿

在任务的server端添加了auth jwt 验证,dtm 未添加对应的 auth header token,在这种情况下,商品库存扣减商品库存扣减补偿都无法成功,dtm进入无限重试商品库存扣减补偿状态。

文档注明会进入 指数退避算法重试 状态,但实际没有,请问是什么原因呢?文档参考

@yedf2
Copy link
Contributor

yedf2 commented Jun 13, 2022

不应当会发生这种情况,会不会是你的微服务框架自动重试了?你开一下dtm的debug日志,诊断一下,如果还有问题,把日志发给我

@mowocc
Copy link
Author

mowocc commented Jun 14, 2022

不应当会发生这种情况,会不会是你的微服务框架自动重试了?你开一下dtm的debug日志,诊断一下,如果还有问题,把日志发给我

在 saga BranchHeaders 添加对应的 auth header token 后,新请求可以成功,未添加的请求还在无限重试中。

微服务使用kratos,使用consul做服务发现,框架没有自动重试,在手动点击 dtm admin ui 的 ForceStop 按钮后,dtm停止了无限重试。

dtm使用docker安装,无限重试时dtm重复打印以下日志:

{"level":"error","ts":"2022-06-14T09:18:17.441Z","caller":"dtmsvr/trans_type_saga.go:130","msg":"exec branch 01 compensate discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert error: http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3.1\n\t/app/dtm/dtmsvr/trans_type_saga.go:130\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:134"}

{"level":"info","ts":"2022-06-14T09:18:17.448Z","caller":"dtmsvr/cron.go:57","msg":"cron job return a trans: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":8,\"next_cron_time\":\"2022-06-14T09:18:27Z\",\"owner\":\"WXF5UvCWhyvaScm4o8a9x9\",\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.463Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert \"CAEQAg==\" result: \"\" err: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngoogle.golang.org/grpc.getChainUnaryInvoker.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:360\ngithub.com/dtm-labs/dtm/dtmsvr.StartSvr.func1\n\t/app/dtm/dtmsvr/svr.go:40\ngoogle.golang.org/grpc.chainUnaryClientInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:348\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:162\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:170\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:182\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:133"}

{"level":"info","ts":"2022-06-14T09:18:17.468Z","caller":"dtmsvr/trans_status.go:36","msg":"TouchCronTime for: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17.4635999Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":16,\"next_cron_time\":\"2022-06-14T09:18:33.4635552Z\",\"owner\":\"WXF5UvCWhyvaScm4o8a9x9\",\"wait_result\":true,\"timeout_to_fail\":2,\"concurrent\":false}"}

@yedf2
Copy link
Contributor

yedf2 commented Jun 14, 2022

给出来的日志太少了,请把重复的内容给出3个重复的

@mowocc
Copy link
Author

mowocc commented Jun 14, 2022

3个重复的


{"level":"error","ts":"2022-06-14T09:18:17.057Z","caller":"dtmsvr/trans_type_saga.go:130","msg":"exec branch 01 compensate discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert error: http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3.1\n\t/app/dtm/dtmsvr/trans_type_saga.go:130\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:134"}

{"level":"info","ts":"2022-06-14T09:18:17.060Z","caller":"dtmsvr/cron.go:57","msg":"cron job return a trans: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":8,\"next_cron_time\":\"2022-06-14T09:18:27Z\",\"owner\":\"jsF7GcPkAAFHwroiG8bPkT\",\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.064Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert \"CAEQAg==\" result: \"\" err: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngoogle.golang.org/grpc.getChainUnaryInvoker.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:360\ngithub.com/dtm-labs/dtm/dtmsvr.StartSvr.func1\n\t/app/dtm/dtmsvr/svr.go:40\ngoogle.golang.org/grpc.chainUnaryClientInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:348\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:162\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:170\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:182\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:133"}

{"level":"info","ts":"2022-06-14T09:18:17.066Z","caller":"dtmsvr/trans_status.go:36","msg":"TouchCronTime for: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17.0642046Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":16,\"next_cron_time\":\"2022-06-14T09:18:33.0641554Z\",\"owner\":\"jsF7GcPkAAFHwroiG8bPkT\",\"wait_result\":true,\"timeout_to_fail\":2,\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.066Z","caller":"dtmsvr/trans_type_saga.go:130","msg":"exec branch 01 compensate discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert error: http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3.1\n\t/app/dtm/dtmsvr/trans_type_saga.go:130\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:134"}

{"level":"info","ts":"2022-06-14T09:18:17.072Z","caller":"dtmsvr/cron.go:57","msg":"cron job return a trans: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":16,\"next_cron_time\":\"2022-06-14T09:18:27Z\",\"owner\":\"qe2EGSqw9A28ugBSQWbYGj\",\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.076Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert \"CAEQAg==\" result: \"\" err: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngoogle.golang.org/grpc.getChainUnaryInvoker.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:360\ngithub.com/dtm-labs/dtm/dtmsvr.StartSvr.func1\n\t/app/dtm/dtmsvr/svr.go:40\ngoogle.golang.org/grpc.chainUnaryClientInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:348\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:162\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:170\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:182\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:133"}

{"level":"info","ts":"2022-06-14T09:18:17.079Z","caller":"dtmsvr/trans_status.go:36","msg":"TouchCronTime for: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17.0770353Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":2,\"next_cron_time\":\"2022-06-14T09:18:19.0769909Z\",\"owner\":\"qe2EGSqw9A28ugBSQWbYGj\",\"wait_result\":true,\"timeout_to_fail\":2,\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.079Z","caller":"dtmsvr/trans_type_saga.go:130","msg":"exec branch 01 compensate discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert error: http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3.1\n\t/app/dtm/dtmsvr/trans_type_saga.go:130\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:134"}

{"level":"info","ts":"2022-06-14T09:18:17.082Z","caller":"dtmsvr/cron.go:57","msg":"cron job return a trans: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":2,\"next_cron_time\":\"2022-06-14T09:18:27Z\",\"owner\":\"6FzqVsivXu6w5x95qHi5CL\",\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.085Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert \"CAEQAg==\" result: \"\" err: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngoogle.golang.org/grpc.getChainUnaryInvoker.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:360\ngithub.com/dtm-labs/dtm/dtmsvr.StartSvr.func1\n\t/app/dtm/dtmsvr/svr.go:40\ngoogle.golang.org/grpc.chainUnaryClientInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:348\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:162\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:170\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:182\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:133"}

{"level":"info","ts":"2022-06-14T09:18:17.089Z","caller":"dtmsvr/trans_status.go:36","msg":"TouchCronTime for: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17.0859737Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":4,\"next_cron_time\":\"2022-06-14T09:18:21.0859295Z\",\"owner\":\"6FzqVsivXu6w5x95qHi5CL\",\"wait_result\":true,\"timeout_to_fail\":2,\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.089Z","caller":"dtmsvr/trans_type_saga.go:130","msg":"exec branch 01 compensate discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert error: http/grpc result should be specified as in:\nhttps://dtm.pub/summary/arch.html#http\nunkown result will be retried: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3.1\n\t/app/dtm/dtmsvr/trans_type_saga.go:130\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:134"}

{"level":"info","ts":"2022-06-14T09:18:17.097Z","caller":"dtmsvr/cron.go:57","msg":"cron job return a trans: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":4,\"next_cron_time\":\"2022-06-14T09:18:27Z\",\"owner\":\"przPyR854v5uUHwuwBP9ZD\",\"concurrent\":false}"}

{"level":"error","ts":"2022-06-14T09:18:17.100Z","caller":"dtmgimp/types.go:46","msg":"grpc client called: discovery://consul-server:8500/shop.goods.service/goods.service.v1.Goods/StockDeductRevert \"CAEQAg==\" result: \"\" err: rpc error: code = Unauthenticated desc = JWT token is missing","stacktrace":"github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcClientLog\n\t/app/dtm/dtmgrpc/dtmgimp/types.go:46\ngoogle.golang.org/grpc.getChainUnaryInvoker.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:360\ngithub.com/dtm-labs/dtm/dtmsvr.StartSvr.func1\n\t/app/dtm/dtmsvr/svr.go:40\ngoogle.golang.org/grpc.chainUnaryClientInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/clientconn.go:348\ngoogle.golang.org/grpc.(*ClientConn).Invoke\n\t/go/pkg/mod/google.golang.org/grpc@v1.45.0/call.go:35\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getURLResult\n\t/app/dtm/dtmsvr/trans_status.go:162\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).getBranchResult\n\t/app/dtm/dtmsvr/trans_status.go:170\ngithub.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).execBranch\n\t/app/dtm/dtmsvr/trans_status.go:182\ngithub.com/dtm-labs/dtm/dtmsvr.(*transSagaProcessor).ProcessOnce.func3\n\t/app/dtm/dtmsvr/trans_type_saga.go:133"}

{"level":"info","ts":"2022-06-14T09:18:17.103Z","caller":"dtmsvr/trans_status.go:36","msg":"TouchCronTime for: {\"ID\":61,\"create_time\":\"2022-06-14T09:18:01Z\",\"update_time\":\"2022-06-14T09:18:17.1010814Z\",\"gid\":\"wJyXGNidcWjQbCxMwjFzt4\",\"trans_type\":\"saga\",\"status\":\"aborting\",\"protocol\":\"grpc\",\"options\":\"{\\\"wait_result\\\":true,\\\"timeout_to_fail\\\":2,\\\"concurrent\\\":false}\",\"next_cron_interval\":8,\"next_cron_time\":\"2022-06-14T09:18:25.1010359Z\",\"owner\":\"przPyR854v5uUHwuwBP9ZD\",\"wait_result\":true,\"timeout_to_fail\":2,\"concurrent\":false}"}

@yedf2
Copy link
Contributor

yedf2 commented Jun 14, 2022

问题在于你的服务器时钟和数据库的时钟对不上,或者时区对不上,只要时钟和时区对上了,这个问题就解决了

@mowocc
Copy link
Author

mowocc commented Jun 15, 2022

问题在于你的服务器时钟和数据库的时钟对不上,或者时区对不上,只要时钟和时区对上了,这个问题就解决了

感谢您的回复,问题原因已经了解清楚,我试图通过修改docker-compose.yaml环境变量修改时区,但未成功。

mysql 运行在 windows 物理机上,时区为+8:00,dtm server 使用 docker compose 启动,进入docker内查看时区为UTC。配置如下:

  dtm:
    image: yedf/dtm
    container_name: dtm
    restart: always
    volumes:
      - ./dtm/config.yaml:/app/dtm/configs/config.yaml
    command:
      - "-c=/app/dtm/configs/config.yaml"
    networks:
      shop_net:
    ports:
      - '36789:36789'
      - '36790:36790'
    environment:
      - TZ=Asia/Shanghai
      - SET_CONTAINER_TIMEZONE=true
      - CONTAINER_TIMEZONE=Asia/Shanghai
      - MICRO_SERVICE_DRIVER=dtm-driver-kratos # 驱动名称
      - MICRO_SERVICE_TARGET=consul://consul-server:8500/dtm # 使用 consul 作为服务发现工具,向 consul-server:8500 地址注册名称为 dtm 的服务
      - MICRO_SERVICE_END_POINT=grpc://192.168.2.43:36790 # 使用 grpc 方式访问 dtm 的地址(consul在docker内部需要发现dtm,同时go微服务运行在物理机上,使用物理机IP兼容两者访问)

config.yaml

Store: 
  Driver: 'mysql'
  Host: '192.168.2.43'
  User: 'root'
  Password: ''
  Port: 3306

如果正式环境使用Kubernetes部署,应该也需要修改时区,请问有解决办法吗?麻烦您了~~

@yedf2
Copy link
Contributor

yedf2 commented Jun 15, 2022

这个你可以问一下你们公司运维,或者问一下熟悉docker的同学。
我对windows的docker领域并不熟悉,到估计是有办法解决的

@mowocc
Copy link
Author

mowocc commented Jun 15, 2022

这个你可以问一下你们公司运维,或者问一下熟悉docker的同学。 我对windows的docker领域并不熟悉,到估计是有办法解决的

再次感谢您一直跟踪回复,谢谢!!已经找公司的运维同学共同寻找解决方法,如果找到了,我会回复在该问题下。

@thful
Copy link

thful commented Jun 17, 2022

这个你可以问一下你们公司运维,或者问一下熟悉docker的同学。我对windows的docker领域并不熟悉,到估计是有办法解决的

再次感谢您一直跟踪回复,谢谢!!已经找公司的运维同学共同寻找解决方法,如果找到了,我会回复在该问题下。

我也遇到了这个问题,因为DTM的官方包无法挂载本地时区,在docker-compose.yml中配置 - /etc/timezone:/etc/timezone:ro 和 - /etc/localtime:/etc/localtime:ro不会生效,

  • 需要docker cp /usr/share/zoneinfo/Asia/Shanghai [容器编号]:/etc/localtime,导入本地时区之后,再保存容器就可以了。

@xyctruth
Copy link
Contributor

xyctruth commented Jun 20, 2022

不管是 docker run , docker-compose , k8s. 都挂载对应时区的 localtime 文件到容器的 /etc/localtime 路径即可

docker run

$ docker run -itd  --name dtm -p 36789:36789 -p 36790:36790 -v /etc/localtime:/etc/localtime  yedf/dtm:latest

docker-compose

dtm:
   volumes:
      - /etc/localtime:/etc/localtime

k8s

创建主机路径卷

volumes:
- name: tz-config
  hostPath:
    path: /etc/localtime

容器挂载卷

volumeMounts:
- name: tz-config
  mountPath: /etc/localtime

@mowocc
Copy link
Author

mowocc commented Jun 22, 2022

@yedf2 @thful @xyctruth 感谢你们的回复,你们的回复给我提供了很好的解决思路。

经过测试,dtm docker 镜像不支持直接设置时区,需要通过挂载时区文件方式设置时区,docker 宿主机为 windows 和 linux 处理方式不同。

dtm-labs/dtm 项目下存在 helper/Dockerfile-release 文件:

# syntax=docker/dockerfile:1
# FROM node:14.19-alpine as builder1
# ARG RELEASE_VERSION
# WORKDIR /app/dtm
# COPY . .
# RUN cd admin && yarn && VITE_ADMIN_VERSION=$RELEASE_VERSION yarn build

FROM --platform=$TARGETPLATFORM golang:1.16-alpine as builder2
ARG TARGETARCH
ARG TARGETOS
ARG RELEASE_VERSION
WORKDIR /app/dtm
# RUN go env -w GOPROXY=https://mirrors.aliyun.com/goproxy/,direct
EXPOSE 8080
COPY . .
# COPY --from=builder1 /app/dtm/admin/dist /app/dtm/admin
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH go build -ldflags="-s -w -X main.Version=$RELEASE_VERSION"

FROM --platform=$TARGETPLATFORM alpine
COPY --from=builder2 /app/dtm/dtm /app/dtm/
WORKDIR /app/dtm
ENTRYPOINT ["/app/dtm/dtm"]

大致逻辑为:dtm 通过 golang:1.16-alpine 镜像打包,在 alpine 镜像中运行 dtm,但 alpine 镜像默认没有时区文件,所以无法直接设置容器时区。

宿主机 linux 和 windows 兼容方案:挂载时区文件后,再设置时区

  • windows 宿主机需要下载 linux时区文件 到对应 docker-compose.yaml 目录

docker run

# 挂载整个时区目录(占用多余空间,不建议)
docker run --name dtm  -p 36789:36789 -p 36790:36790 -e TZ=Asia/Shanghai -v /usr/share/zoneinfo:/usr/share/zoneinfo -itd yedf/dtm
# 只挂载需要的时区(占用空间小,推荐使用)
docker run --name dtm  -p 36789:36789 -p 36790:36790 -e TZ=Asia/Shanghai -v /usr/share/zoneinfo/Asia/Shanghai:/usr/share/zoneinfo/Asia/Shanghai -itd yedf/dtm

docker-compose

version: '3'

services:

  dtm:
    image: yedf/dtm
    container_name: dtm
    restart: always
    volumes:
      - ./dtm/zoneinfo/Asia/Shanghai:/usr/share/zoneinfo/Asia/Shanghai # 挂载时区文件
      - ./dtm/config.yaml:/app/dtm/configs/config.yaml
    command:
      - "-c=/app/dtm/configs/config.yaml"
    networks:
      shop_net:
    ports:
      - '36789:36789'
      - '36790:36790'
    environment:
      - TZ=Asia/Shanghai # 设置时区

networks:
  shop_net:
    driver: bridge

宿主机只适用于 linux 方案:挂载为系统时间

docker run

docker run --name dtm  -p 36789:36789 -p 36790:36790 -v /usr/share/zoneinfo/Asia/Shanghai:/etc/localtime -itd yedf/dtm

docker-compose

version: '3'

services:

  dtm:
    image: yedf/dtm
    container_name: dtm
    restart: always
    volumes:
      - ./dtm/zoneinfo/Asia/Shanghai:/etc/localtime # 挂载为系统时间
      - ./dtm/config.yaml:/app/dtm/configs/config.yaml
    command:
      - "-c=/app/dtm/configs/config.yaml"
    networks:
      shop_net:
    ports:
      - '36789:36789'
      - '36790:36790'

networks:
  shop_net:
    driver: bridge

宿主机为 windows 环境测试后发现:直接挂载时区文件 ./dtm/zoneinfo/Asia/Shanghai:/etc/localtime 到系统时间,无法修改容器时区。

Kubernetes 还未测试,大致和 docker-compose 方式相同。

@yedf2
Copy link
Contributor

yedf2 commented Jun 22, 2022

最新版本v1.14.3不再要求 dtm的时区与数据库一致:
60abdfc

@mowocc
Copy link
Author

mowocc commented Jun 23, 2022

最新版本v1.14.3不再要求 dtm的时区与数据库一致: 60abdfc

感谢及时更新,但 dtm server 更新到 1.14.3 后,saga 事务报以下错误,还原到 1.14.2 后,可以正常使用。

 code=503 reason= stack=rpc error: code = Unavailable desc = error reading from server: EOF 

dtm 1.14.3 版本处理事务时,子事务没有收到事务请求消息,dtm 日志如下:

{"level":"info","ts":"2022-06-23T12:41:08.803+0800","caller":"dtmgimp/types.go:33","msg":" 0ms <nil> /dtmgimp.Dtm/NewGid {\"Gid\":\"RTnESgYpzgE8QJRqCnnmNj\"} {}"}

panic: Error 1054: Unknown column 'rollback_reason' in 'field list'


goroutine 26 [running]:

github.com/dtm-labs/dtm/dtmutil.(*tracePlugin).Initialize.func2(0xc00079e000)

/app/dtm/dtmutil/db.go:71 +0x33e

gorm.io/gorm.(*processor).Execute(0xc0003205a0, 0xc00079e000, 0x22e19c2)

/go/pkg/mod/gorm.io/gorm@v1.22.2/callbacks.go:130 +0x3f5

gorm.io/gorm.(*DB).Create(0xc00079e000, 0x1fd95c0, 0xc00067d6c0, 0x1)

/go/pkg/mod/gorm.io/gorm@v1.22.2/finisher_api.go:24 +0xc5

github.com/dtm-labs/dtm/dtmsvr/storage/sql.(*Store).MaySaveNewTrans.func1(0xc0002e91a0, 0x0, 0x0)

/app/dtm/dtmsvr/storage/sql/sql.go:105 +0x149

gorm.io/gorm.(*DB).Transaction(0xc00043e7e0, 0xc000bd2558, 0x0, 0x0, 0x0, 0x0, 0x0)

/go/pkg/mod/gorm.io/gorm@v1.22.2/finisher_api.go:548 +0x3da

github.com/dtm-labs/dtm/dtmsvr/storage/sql.(*Store).MaySaveNewTrans(0x3698da0, 0xc00067d6c0, 0xc0003dd680, 0x6, 0x8, 0x22d9362, 0x4)

/app/dtm/dtmsvr/storage/sql/sql.go:101 +0xd6

github.com/dtm-labs/dtm/dtmsvr.(*TransGlobal).saveNew(0xc00067d6c0, 0x0, 0x0, 0x0, 0x0, 0x0)

/app/dtm/dtmsvr/trans_process.go:91 +0x34e

github.com/dtm-labs/dtm/dtmsvr.svcSubmit(0xc00067d6c0, 0xc0000ff410, 0xc0002c1400)

/app/dtm/dtmsvr/api.go:23 +0x5d

github.com/dtm-labs/dtm/dtmsvr.(*dtmServer).Submit(0x3698da0, 0x2780298, 0xc0000ff410, 0xc0002c1400, 0x3698da0, 0x0, 0x0)

/app/dtm/dtmsvr/api_grpc.go:28 +0x53

github.com/dtm-labs/dtm/dtmgrpc/dtmgpb._Dtm_Submit_Handler.func1(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0x2, 0x2, 0x1f1e9a0, 0xc000959988)

/app/dtm/dtmgrpc/dtmgpb/dtmgimp_grpc.pb.go:161 +0x89

github.com/dtm-labs/dtm/dtmgrpc/dtmgimp.GrpcServerLog(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0xc0001554e0, 0xc000bd2258, 0x49ba06, 0x62b3eee4, 0x2ff1a070, 0x1aaaf1b623ec)

/app/dtm/dtmgrpc/dtmgimp/types.go:27 +0x1d6

google.golang.org/grpc.chainUnaryInterceptors.func1.1(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0x203000, 0x0, 0x0, 0x7f9b2bcedb98)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:1117 +0x8c

github.com/dtm-labs/dtm/dtmsvr.grpcMetrics(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0xc0001554e0, 0xc000611180, 0x0, 0x0, 0x0, 0x0)

/app/dtm/dtmsvr/metrics.go:86 +0x182

google.golang.org/grpc.chainUnaryInterceptors.func1.1(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0xc000959b48, 0x40e398, 0x18, 0x20539c0)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:1120 +0x11c

google.golang.org/grpc.chainUnaryInterceptors.func1(0x2780298, 0xc0000ff410, 0x218a5c0, 0xc0002c1400, 0xc0001554e0, 0xc000bd2258, 0xc000835bb8, 0x51d766, 0x214d1e0, 0xc0000ff410)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:1122 +0xeb

github.com/dtm-labs/dtm/dtmgrpc/dtmgpb._Dtm_Submit_Handler(0x20bb360, 0x3698da0, 0x2780298, 0xc0000ff410, 0xc0006b3140, 0xc0002da6a0, 0x2780298, 0xc0000ff410, 0xc000010380, 0x335)

/app/dtm/dtmgrpc/dtmgpb/dtmgimp_grpc.pb.go:163 +0x150

google.golang.org/grpc.(*Server).processUnaryRPC(0xc000880000, 0x279a278, 0xc00067cd00, 0xc00017b8c0, 0xc00043e720, 0x3649f38, 0x0, 0x0, 0x0)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:1283 +0x544

google.golang.org/grpc.(*Server).handleStream(0xc000880000, 0x279a278, 0xc00067cd00, 0xc00017b8c0, 0x0)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:1620 +0xd0c

google.golang.org/grpc.(*Server).serveStreams.func1.2(0xc0008ba620, 0xc000880000, 0x279a278, 0xc00067cd00, 0xc00017b8c0)

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:922 +0xab

created by google.golang.org/grpc.(*Server).serveStreams.func1

/go/pkg/mod/google.golang.org/grpc@v1.46.2/server.go:920 +0x1fd

{"level":"info","ts":"2022-06-23T12:41:09.439+0800","caller":"maxprocs/maxprocs.go:47","msg":"maxprocs: Leaving GOMAXPROCS=6: CPU quota undefined"}

{"level":"info","ts":"2022-06-23T12:41:09.439+0800","caller":"dtmimp/utils.go:175","msg":"opening standalone mysql: ****root:@tcp(192.168.2.43:3306)/dtm?charset=utf8mb4&parseTime=true&loc=Local&interpolateParams=true"}

{"level":"info","ts":"2022-06-23T12:41:09.443+0800","caller":"dtmsvr/svr.go:29","msg":"start dtmsvr"}

{"level":"info","ts":"2022-06-23T12:41:09.444+0800","caller":"dtmsvr/svr.go:48","msg":"dtmsvr http listen at: 36789"}

{"level":"info","ts":"2022-06-23T12:41:09.444+0800","caller":"dtmsvr/svr.go:61","msg":"grpc listening at [::]:36790"}

{"level":"info","ts":"2022-06-23T12:41:09.545+0800","caller":"dtmsvr/svr.go:74","msg":"RegisterGrpcService: dtm-driver-kratos"}

{"level":"info","ts":"2022-06-23T12:41:09.569+0800","caller":"dtmsvr/svr.go:81","msg":"RegisterHTTPService: default"}

{"level":"info","ts":"2022-06-23T12:41:09.569+0800","caller":"dtm/main.go:70","msg":"admin is served from dir 'admin/dist/'"}

{"level":"info","ts":"2022-06-23T12:41:09.569+0800","caller":"dtm/main.go:83","msg":"admin is running at: http://localhost:36789"}

{"level":"info","ts":"2022-06-23T12:41:09.569+0800","caller":"dtmutil/db.go:103","msg":"connecting 'mysql' '192.168.2.43' 'root' '3306' 'dtm'"}

@yedf2
Copy link
Contributor

yedf2 commented Jun 23, 2022

表里面加了字段,fail_reason,加一下就行

@mowocc
Copy link
Author

mowocc commented Jun 23, 2022

表里面加了字段,fail_reason,加一下就行

添加 rollback_reason 字段后可以正常运行。

经过测试,dtm 1.14.3 版本与 mysql 时区不一致情况下,可以正常进入 指数退避算法重试 状态,问题已从 dtm 程序层面解决。

感谢作者对 dtm 的更新,以及问题的及时回复,dtm 是一个优秀的作品,希望您能不断的完善 dtm 体系,成长为一个健壮的产品,成为微服务体系不可或缺的一环。

@mowocc mowocc closed this as completed Jun 27, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants