logs: cockroach check debug-check-config does not match user created yaml configuration

[thtruo]

Observation

I have a logs.yaml configuration that looks like this:

file-defaults:
  dir: cockroach-data/logs 
  filter: INFO             
  redact: false            
  redactable: false      
  max-file-size: 10MB 
  max-group-size: 100MB  
  format: json             
sinks:
 file-groups:              
   security:               # cockroach-security.log
     channels: "SESSIONS, USER_ADMIN, PRIVILEGES, SENSITIVE_ACCESS"
     format: json
   health:                 # cockroach-health.log
     channels: "HEALTH, OPS"
     format: json
   devonly:                # cockroach-devonly.log 👈
     channels: "DEV"
     format: json
 stderr:
   channels: "DEV"         # default is ALL
   format: json
   filter: INFO            # filter: NONE disables stderr by default (this is the default config)

Note that I have a line that only pushes DEV channel events into a file called cockroach-devonly.log.

However, cockroach check debug-check-config --log="$(cat logs.yaml)" looks like this:

Expectation
I expected only the DEV channel events would go into cockroach-devonly.log. It caught me off guard to see SQL_* and STORAGE channel events go into cockroach-devonly.log. My YAML configuration explicitly includes one channel (DEV).

cc @knz again let me know if I'm making a user error or if this is indeed expected
cc @piyush-singh for awareness

Jira issue: CRDB-3340

[knz]

The current logic automatically takes all the channels not otherwise listed by the user's configuration, and redirects them to whatever sink the user configured for the DEV channel.

The motivation behind this choice is to have a natural "catch-all" whenever we add new channels in CockroachDB in the future, to avoid having these entries lost altogether.

I think I understand the confusion however. What behavior would you suggest instead, that reduces the confusion but also keeps a "catch-all" behavior for future, yet-unknown channels?

[thtruo]

The current logic automatically takes all the channels not otherwise listed by the user's configuration, and redirects them to whatever sink the user configured for the DEV channel.

Ohh! Understood - I didn't pick up on that condition in my first go-around. That's clear now.

I get the motivation for having a "catch-all". My first impression was that it felt unnatural that unlisted channels get redirected to the same sink as the DEV channel, but I also get it. Definitely worth clarifying and calling out in docs cc @taroface

The only alternative behavior that comes to mind is making DEV channel events go into its own file unless it is listed as a channel in another sink. In a sense, DEV channel events will be like "stray errors"; where "stray errors" go into their own cockroach-stderr.log, DEV channel events go into their own cockroach-dev.log.

So if we follow that line of thinking, scenarios will play out so that unlisted channels will end up in the same "catchall" file. And then DEV ends up in its own file unless it is explicitly listed for another sink.

I think the merits of having DEV channel events go into their own file is to emphasize the unique characteristics of it, as you called out in your RFC:

• The channel is used during CockroachDB development
• This channel is special in that there are no constraints as to what may or may not be logged on it
• These events are undocumented
• And in this best practice section: do not centrally collect logs from the DEV channel ahead of scraping logs for data-mining purposes

Let me know if this suggestion runs into a blindspot. If it doesn't make sense to deviate from the current logic, that's okay too. Either way, it reinforces the need to clearly call this out in docs.