[Feature request] Wait a while for Bitcoin Core to start instead of crashing immediately

[NicolasDorier]

At startup, c-lightning should wait a bit for Bitcoin core to start and not crash (#4352)

[NicolasDorier]

Workaround we use in our fork: btcpayserver@2ef9f6f

[laanwj]

I have had a similar issue. You might want to look into bitcoin/bitcoin#21007 as for fixing this from the Bitcoin Core side.

[NicolasDorier]

I do not think this is related, I am not running bitcoind as a deamon. The error is that connection to bitcoind is refused when clightning try to connect to it.

In my case, bitcoind stays in the foreground in its container, and docker is the one starting services depending on it (clightning)

I think docker is using some shaky heuristic to know when it is the right time to launch a dependant services.
Somehow, the fact that I need to use bitcoin-wallet create in bitcoin core's container before starting bitcoin core, made clightning starts too early. (before rpc bind to the port)

[laanwj]

It's related in the sense that it makes for an easy way to know when something dependent on bitcoind can be started. The -daemonwait process will only return when either the initialization is complete and RPC can process commands, or when initialization failed.

I don't know anything about docker though.

[darosior]

We do wait for bitcoind on startup indefinitely, if it its at least on warmup:

lightning/plugins/bcli.c

Lines 817 to 871 in 483579f

	static void wait_and_check_bitcoind(struct plugin *p)
	{
	int from, status, ret;
	pid_t child;
	const char **cmd = gather_args(bitcoind, "getnetworkinfo", NULL);
	bool printed = false;
	char *output = NULL;
	for (;;) {
	tal_free(output);
	child = pipecmdarr(NULL, &from, &from, cast_const2(char **,cmd));
	if (child < 0) {
	if (errno == ENOENT)
	bitcoind_failure(p, "bitcoin-cli not found. Is bitcoin-cli "
	"(part of Bitcoin Core) available in your PATH?");
	plugin_err(p, "%s exec failed: %s", cmd[0], strerror(errno));
	}
	output = grab_fd(cmd, from);
	while ((ret = waitpid(child, &status, 0)) < 0 && errno == EINTR);
	if (ret != child)
	bitcoind_failure(p, tal_fmt(bitcoind, "Waiting for %s: %s",
	cmd[0], strerror(errno)));
	if (!WIFEXITED(status))
	bitcoind_failure(p, tal_fmt(bitcoind, "Death of %s: signal %i",
	cmd[0], WTERMSIG(status)));
	if (WEXITSTATUS(status) == 0)
	break;
	/* bitcoin/src/rpc/protocol.h:
	* RPC_IN_WARMUP = -28, //!< Client still warming up
	*/
	if (WEXITSTATUS(status) != 28) {
	if (WEXITSTATUS(status) == 1)
	bitcoind_failure(p, "Could not connect to bitcoind using"
	" bitcoin-cli. Is bitcoind running?");
	bitcoind_failure(p, tal_fmt(bitcoind, "%s exited with code %i: %s",
	cmd[0], WEXITSTATUS(status), output));
	}
	if (!printed) {
	plugin_log(p, LOG_UNUSUAL,
	"Waiting for bitcoind to warm up...");
	printed = true;
	}
	sleep(1);
	}
	parse_getnetworkinfo_result(p, output);
	tal_free(cmd);
	}

From your error in #4352 (comment) it looks like bitcoind has not even be started yet, and i'm not sure it's reasonable to wait with no clue that a bitcoind is actually going to start behind us at some point?

[NicolasDorier]

@darosior it would be nice to at least wait for 5-10 sec. (actually even 1 would be enough)

That's what our clightning fork is doing btcpayserver@2ef9f6f

As I said, the problem is that even if we configure c-lightning to depends on bitcoind at docker level, the fact that in bitcoin 0.21.0 I need to call bitcoin-create make docker start c-lightning start too early.

There is ways to work around that (like my commit above), or by having some boilerplate code in the docker entrypoint, but then waiting for the warmup become useless, as it is done in the bash script anyway.

[NicolasDorier]

@laanwj sadly, docker does not care about when bitcoind returns before starting the dependant services. And actually, bitcoind must run in foreground if hosted in container, so I can't even use -daemonwait

[cdecker]

One option might be to expose the -rpcwait command line option
inside of the bcli plugin. That way we'd be waiting both for the RPC
to startup, and for the warmup to complete.

-rpcwait internally polls the RPC interface ever 1 second
indefinitely, and doesn't fail if the destination is unreachable,
which is the issue at hand if I see correctly. So starting docker
containers in any order would just have the lightningd container
wait for the bitcoind container to become available.

My main concern is that it might mask some of the errors indefinitely,
like a misconfigured RPC location or credentials, would no longer
report an error, rather we'd just sit there and wait forever. Might be
a good idea to propose an optional numeric argument to -rpcwait
indicating the number of seconds to wait before reporting failure. So
-rpcwait=10 would try 10 times, and after 10 seconds, we'd receive
the error message which we could then report, alerting the operator of
the issue.

@laanwj do you think that such a proposal would be welcome in
bitcoin/bitcoin, or is there prior discussion on this?

[laanwj]

@laanwj do you think that such a proposal would be welcome in bitcoin/bitcoin, or is there prior discussion on this?

I'm not aware of any prior discussion of this. But sounds good to me.

[NicolasDorier]

@cdecker I tried to add -rpcwait, but this did not seem to solve the issue in my case. I think rpcwait does not work if the server refuse connection, it just returns instantly.

My main concern is that it might mask some of the errors indefinitely,
like a misconfigured RPC location or credentials, would no longer
report an error, rather we'd just sit there and wait forever.

Just try for 20 sec. You can detect whether rpc is loading or if it fails to connect. If rpc loading -> wait indefinitely. If fails to connect -> fails after 20sec or so.

[cdecker]

Are you sure about that? I just tried with bitcoin-cli pointing at a non-existent server (port that isn't being served on my local machine) and it hung indefinitely. So unless there is some weirdness with bitcoind binding to the port but not accepting connections I don't see how that could fail.

Furthermore if I then start a bitcoind on that port it works as expected. Can you retry in a dockerized setting?

[cdecker]

Well, I got curious myself what'd happen, so here's a gist that does a minimal recreation: https://gist.github.com/cdecker/4bdcf21192855022a02bd895d06fd18f

It starts bitcoind in one container, starts bitcoin-cli in another called lightningd and then attempts to call echo from the latter. If I remove the -rpcwait argument in the bitcoin-cli invokation it indeed complains, if I add it, all is good. To exacerbate the issue I also added a 10 second delay to the bitcoind startup, again no problem.

So my proposal would be to file a PR adding the max wait time to -rpcwait and then pass rpcwait through to the bcli plugin.

[NicolasDorier]

@cdecker I am not sure. I tried to use rpcwait, locally and as you pointed out, it was hanging. However, before trying my current solution, I tried inserting rpcwait directly in the gather_args function, and it did not worked.

Maybe it is because the bitcoin-cli we are using in the docker is rather old (0.18). Though I checked it had support for rpcwait.

[NicolasDorier]

The docker-compose we use is here:

https://github.com/btcpayserver/BTCPayServer.Lightning/blob/master/tests/docker-compose.yml

btcpayserver/lightning:v0.9.3-1-dev is with my workaround.

You can easily test what happen by:

1. Make your change locally
2. Create a dummy commit (you need to commit because the docker build is cloning the repo)
3. docker build -t btcpayserver/lightning:v0.9.3-1-dev .
4. docker-compose down -v
5. docker-compose up -d
6. docker logs <lightning_container>

the docker-compose down -v make sure that the bug will be reproduced, as it makes sure bitcoind will call bitcoin-wallet create when starting, which exposed the bug.

Note that sometimes it works, so you need another round of down/up.