Out of space no longer logged when a node runs out of space

[RichardoC]

Describe the problem

Since upgrading to docker image cockroachdb/cockroach:v21.1.3 from cockroachdb/cockroach:v20.2.1 cockroach will not log to stderr that it had run out of space before exiting upon a restart.

To Reproduce

Run cockroachdb/cockroach:v21.1.3 with entrypoint
cockroach start-single-node --store /storage/ --insecure --log="sinks: {stderr: {no-color: true, channels: [ALL]}, file-groups: {default: {channels: [All]}}}"
Where /storage/ points to a volume with no space left

See that nothing is logged to stderr

Expected behavior
Run cockroachdb/cockroach:v20.2.1 with entrypoint
cockroach start-single-node --store /storage/ --insecure
Where /storage/ points to a volume with no space left

Logs from cockroachdb/cockroach:v20.2.1

E210702 15:35:36.915758 1 util/log/exit_override.go:59 ⋮ logging error: write ‹/storage/logs/cockroach.database-service-559d8ccd44-9xsz7.unknownuser.2021-07-02T15_34_39Z.000001.log›: no space left on device

Environment:

• CockroachDB version
  docker image cockroachdb/cockroach:v21.1.3
• Server OS:
  Ubuntu 18.04
• Client app [e.g. cockroach sql, JDBC, ...]
  N/A

Additional context
What was the impact?
Makes it more difficult to work out why a cockroach node won't start.

Jira issue: CRDB-8417

[blathers-crl]

Hello, I am Blathers. I am here to help you get the issue triaged.

Hoot - a bug! Though bugs are the bane of my existence, rest assured the wretched thing will get the best of care here.

I was unable to automatically find someone to ping.

If we have not gotten back to your issue within a few business days, you can try the following:

• Join our community slack channel and ask on #cockroachdb.
• Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is otan.}

[rafiss]

Thanks for reporting this @RichardoC !

cc @knz for triaging

[RichardoC]

Hi there, any updates?

[knz]

Not yet, but thanks for the reminder.

[knz]

cc @piyush-singh @jtsiros for prioritization

[jtsiros]

Right now, we are targeting this for after the new year given the December milestone priorities. Please ping @mwang1026 if this becomes urgent.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
10 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[RichardoC]

/remove-label stale