sysbox-fs error, context deadline exceeded: unknown

[myugan]

This is an inconsistent issue but can be fixed by restarting the sysbox service, this issue causes docker in Created state and I checked with docker inspect to see what's the issue, then here it is:

{
        "Id": "d3dcf20ad2f3f1287f9f44ff5d986cf4335044c90533bc85e42e68a4ac41ce4f",
        "Created": "2021-04-30T22:53:56.734536333Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "bin/bash"
        ],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 128,
            "Error": "OCI runtime create failed: failed to pre-register with sysbox-fs: failed to register container with sysbox-fs: rpc error: code = DeadlineExceeded desc = context deadline exceeded: unknown",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
...

[rodnymolina]

@myugan, thanks for reporting this one.

I believe this can be a duplicate of issue #266 on which i'm currently working on. Just a few hours ago i merged some changes that mitigate (but not fully correct) this issue. I think i'm close to identify a complete fix for all similar issues.

[rodnymolina]

@myugan, could you please upload the sysbox-fs.log corresponding to the timestamp where you saw the issue last time?

[myugan]

here it is, the log that related to a container:

time="2021-04-30 22:47:35" level=info msg="Container unregistration completed: id = 6b07069476814c6243b5a1d5e4949370038459e5a72e781daea5bbb6fd06331a"
time="2021-04-30 22:52:04" level=info msg="Container unregistration completed: id = 5c8c09915663e64c7119551236bdd0b302ccd245026a27564bb1cd03954b1eb3"
time="2021-04-30 22:52:04" level=info msg="Container unregistration completed: id = 6f15bf40175a5ccade8ced1e567f69f42faa11c49ed9c4e1af15941c5b28474f"
time="2021-04-30 22:52:04" level=info msg="Container unregistration completed: id = db2eacc50a4705810b845695cf5c21c7f3705052228328ddcbebef369dd8290f"
time="2021-04-30 22:53:25" level=warning msg="Sysbox-fs first child process error status: pid = 23955"
time="2021-04-30 22:53:25" level=warning msg="Sysbox-fs first child process error status: pid = 23966"
time="2021-04-30 22:53:25" level=warning msg="Sysbox-fs first child process error status: pid = 24004"
time="2021-04-30 22:53:26" level=info msg="reaper: reaped pid 23956"
time="2021-04-30 22:53:26" level=info msg="reaper: reaped pid 23967"
time="2021-04-30 22:53:26" level=info msg="reaper: reaped pid 24005"
time="2021-04-30 22:53:26" level=info msg="reaper: nothing to reap"
time="2021-04-30 22:57:24" level=warning msg="Sysbox-fs first child process error status: pid = 28386"
time="2021-04-30 22:57:24" level=warning msg="Sysbox-fs first child process error status: pid = 28412"
time="2021-04-30 22:57:24" level=warning msg="Sysbox-fs first child process error status: pid = 28455"
time="2021-04-30 22:57:25" level=info msg="reaper: reaped pid 28387"
time="2021-04-30 22:57:25" level=info msg="reaper: reaped pid 28413"
time="2021-04-30 22:57:25" level=info msg="reaper: reaped pid 28457"
time="2021-04-30 22:57:25" level=info msg="reaper: nothing to reap"

[rodnymolina]

Thanks @myugan. I think this one is probably a dup of issue #266, for which I believe I have already identified two separate fixes (the first one was already merged a few days ago, and the second one is on its way).

Let's keep this issue open till you have a chance to verify both fixes.

[rodnymolina]

@myugan, I believe this issue has been fixed by the work we've done as part of #266, so I'll go ahead and close this one now. Please re-open it if you reproduce it.