远程缓存会卡死 #2512

Domain · 2022-06-29T13:15:36Z

Xmake 版本

2.6.8

操作系统版本和架构

Windows 11

描述问题

服务器开启了远程缓存，客户端找到了缓存，但是没法使用，一直显示pull cache，超过1个多小时都不会超时。服务器有报错：

C:\Users\build>xmake service -vD
: listening 0.0.0.0:9691 ..
: listening 0.0.0.0:9692 ..
: listening 0.0.0.0:9693 ..
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : session connected
: : : on handle message(9)
{
  token = "caa00ef84ecf37b842326e71137db005",
  session_id = "73885b64",
  code = 9,
  filename = "4f505cbcd68cb57263f9407e5243d899"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:872]: in function 'raiselevel'
    [@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
    [@programdir\modules\private\service\remote_cache\server.lua:88]:

: : : send ok
: : : on handle message(9)
{
  token = "caa00ef84ecf37b842326e71137db005",
  session_id = "73885b64",
  code = 9,
  filename = "5fbf8875ae9048b147b097bf7c22398c"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:872]: in function 'raiselevel'
    [@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
    [@programdir\modules\private\service\remote_cache\server.lua:88]:

: : : send ok
: : : on handle message(9)
{
  token = "caa00ef84ecf37b842326e71137db005",
  session_id = "73885b64",
  code = 9,
  filename = "e75cc824dba5061ca15dcbc61e7a1142"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
    [C]: in function 'error'
    [@programdir\core\base\os.lua:872]: in function 'raiselevel'
    [@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
    [@programdir\modules\private\service\remote_cache\server.lua:88]:

: : : send ok
: : : on handle message(9)
{
  token = "caa00ef84ecf37b842326e71137db005",
  session_id = "73885b64",
  code = 9,
  filename = "f63a9bda6bcd53499c3bae4fe3320436"
}

期待的结果

能正常获取缓存，或者快速超时，转本地缓存

工程配置

No response

附加信息和错误日志

No response

Domain · 2022-06-29T13:17:16Z

C:\Users\build>xmake service -vD
<remote_build_server>: listening 0.0.0.0:9691 ..
<remote_cache_server>: listening 0.0.0.0:9692 ..
<distcc_build_server>: listening 0.0.0.0:9693 ..
<remote_cache_server>: <socket: tcp4/872.0>: session connected
<remote_cache_server>: <socket: tcp4/908.0>: session connected
<remote_cache_server>: <socket: tcp4/912.0>: session connected
<remote_cache_server>: <socket: tcp4/916.0>: session connected
<remote_cache_server>: <socket: tcp4/920.0>: session connected
<remote_cache_server>: <socket: tcp4/924.0>: session connected
<remote_cache_server>: <socket: tcp4/928.0>: session connected
<remote_cache_server>: <socket: tcp4/932.0>: session connected
<remote_cache_server>: <socket: tcp4/936.0>: session connected
<remote_cache_server>: <socket: tcp4/940.0>: session connected
<remote_cache_server>: <socket: tcp4/944.0>: session connected
<remote_cache_server>: <socket: tcp4/948.0>: session connected
<remote_cache_server>: <socket: tcp4/952.0>: session connected
<remote_cache_server>: <socket: tcp4/956.0>: session connected
<remote_cache_server>: <socket: tcp4/960.0>: session connected
<remote_cache_server>: <socket: tcp4/964.0>: session connected
<remote_cache_server>: <socket: tcp4/968.0>: session connected
<remote_cache_server>: <socket: tcp4/972.0>: session connected
<remote_cache_server>: <socket: tcp4/976.0>: session connected
<remote_cache_server>: <socket: tcp4/980.0>: session connected
<remote_cache_server>: <socket: tcp4/984.0>: session connected
<remote_cache_server>: <socket: tcp4/988.0>: session connected
<remote_cache_server>: <socket: tcp4/992.0>: session connected
<remote_cache_server>: <socket: tcp4/996.0>: session connected
<remote_cache_server>: <socket: tcp4/1000.0>: session connected
<remote_cache_server>: <socket: tcp4/1004.0>: session connected
<remote_cache_server>: <socket: tcp4/1008.0>: session connected
<remote_cache_server>: <socket: tcp4/1012.0>: session connected
<remote_cache_server>: <socket: tcp4/1016.0>: session connected
<remote_cache_server>: <socket: tcp4/1020.0>: session connected
<remote_cache_server>: <socket: tcp4/1028.0>: session connected
<remote_cache_server>: <socket: tcp4/872.0>: <session 73885b64>: on handle message(9)
{
token = "caa00ef84ecf37b842326e71137db005",
session_id = "73885b64",
code = 9,
filename = "4f505cbcd68cb57263f9407e5243d899"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
[C]: in function 'error'
[@programdir\core\base\os.lua:872]: in function 'raiselevel'
[@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
[@programdir\modules\private\service\remote_cache\server.lua:88]:

<remote_cache_server>: <socket: tcp4/872.0>: <session 73885b64>: send ok
<remote_cache_server>: <socket: tcp4/908.0>: <session 73885b64>: on handle message(9)
{
token = "caa00ef84ecf37b842326e71137db005",
session_id = "73885b64",
code = 9,
filename = "5fbf8875ae9048b147b097bf7c22398c"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
[C]: in function 'error'
[@programdir\core\base\os.lua:872]: in function 'raiselevel'
[@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
[@programdir\modules\private\service\remote_cache\server.lua:88]:

<remote_cache_server>: <socket: tcp4/908.0>: <session 73885b64>: send ok
<remote_cache_server>: <socket: tcp4/912.0>: <session 73885b64>: on handle message(9)
{
token = "caa00ef84ecf37b842326e71137db005",
session_id = "73885b64",
code = 9,
filename = "e75cc824dba5061ca15dcbc61e7a1142"
}

@programdir\modules\private\service\remote_cache\server.lua:88: session has not been connected!
stack traceback:
[C]: in function 'error'
[@programdir\core\base\os.lua:872]: in function 'raiselevel'
[@programdir\core\sandbox\modules\utils.lua:181]: in function 'assert'
[@programdir\modules\private\service\remote_cache\server.lua:88]:

<remote_cache_server>: <socket: tcp4/912.0>: <session 73885b64>: send ok
<remote_cache_server>: <socket: tcp4/916.0>: <session 73885b64>: on handle message(9)
{
token = "caa00ef84ecf37b842326e71137db005",
session_id = "73885b64",
code = 9,
filename = "f63a9bda6bcd53499c3bae4fe3320436"
}

waruqi · 2022-06-30T02:32:13Z

这两天没时间细看，等周末看下

waruqi · 2022-06-30T09:53:03Z

当前 remote cache client 传给 server 的 73885b64 变了，导致 server 端找不到之前已有的 session 。。

要么你删了 .xmake 然后 xmake service --connect --ccache --distcc 重连下，client 会生成新的 session id 给 server ，两端刷下再试试。。如果还是不行，保留两端完整 -vD logs 都发下

Domain · 2022-07-04T02:05:58Z

几天没跑了，我没有删除.xmake，现在只是重新connect，没问题了。我觉得是不是加一个超时好一点？出问题了自动不用缓存

waruqi · 2022-07-05T07:18:52Z

超时我初步支持了，可以切到 timeout 分支试下。。xmake update -s github:xmake-io/xmake#timeout

$ cat ~/.xmake/service/client.conf
{
    send_timeout = 5000,
    recv_timeout = 5000,
    connect_timeout = 5000,
    distcc_build = {
        send_timeout = 5000,
        recv_timeout = 5000,
        connect_timeout = 5000,
    },
    remote_cache = {
        send_timeout = 5000,
        recv_timeout = 5000,
        connect_timeout = 5000,
    }
}

默认配置，timeout 都是 -1，你可以可选配置上对应的 send/recv/connect 超时，也可以在 distcc_build/remote_cache 的配置块里面，单独对某个 client 服务配置上。

waruqi · 2022-07-05T10:31:18Z

我merge了后续有问题先关了再开

Domain added the bug label Jun 29, 2022

waruqi mentioned this issue Jul 5, 2022

add timeout for service #2530

Merged

waruqi added this to the v2.6.9 milestone Jul 5, 2022

waruqi closed this as completed Jul 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

远程缓存会卡死 #2512

远程缓存会卡死 #2512

Domain commented Jun 29, 2022

Domain commented Jun 29, 2022

waruqi commented Jun 30, 2022

waruqi commented Jun 30, 2022

Domain commented Jul 4, 2022

waruqi commented Jul 5, 2022 •

edited

Loading

waruqi commented Jul 5, 2022

远程缓存会卡死 #2512

远程缓存会卡死 #2512

Comments

Domain commented Jun 29, 2022

Xmake 版本

操作系统版本和架构

描述问题

期待的结果

工程配置

附加信息和错误日志

Domain commented Jun 29, 2022

waruqi commented Jun 30, 2022

waruqi commented Jun 30, 2022

Domain commented Jul 4, 2022

waruqi commented Jul 5, 2022 • edited Loading

waruqi commented Jul 5, 2022

waruqi commented Jul 5, 2022 •

edited

Loading