feat: provide logger customization

[antongolub]

closes #376
closes #405

• Tests pass
• Appropriate changes to README are included in PR
• Types updated

test('logger works', async () => {
  let stdout = ''
  let stderr = ''
  $.logFormat = (msg) => msg.map((m) => m.toUpperCase())
  $.logPrint = (data, err) => {
    if (data) stdout += data
    if (err) stderr += err
  }
  $.logIgnore = ['b*', 'fetch']
  // NOTE first arg declares scope
  log('foo', 'foo-test')
  log('bar', 'bar-test')
  log('baz', 'baz-test')
  log('fetch', 'example.com')

  assert(stdout.includes('FOO-TEST'))
  assert(!stdout.includes('BAR-TEST'))
  assert(!stdout.includes('BAZ-TEST'))
  assert(!stdout.includes('EXAMPLE.COM'))

  $.logOutput = 'stderr'
  log('foo', 'foo')
  assert(stderr.includes('FOO'))

  delete $.logPrint
  delete $.logFormat
  delete $.logIgnore
  delete $.logOutput
})

ignore dep already comes with globby.

Pros

• filtering events by their scope: cd/fetch/cmd,...
• single entry point to introduce creds concealer
• custom formatting to attach mdc, timestamps, etc
• logging rules are attached to current ctx, so they can be applied to a cmd block.
• applied to spawned processes outputs
• if (!verbose) return at single place

Cons

• inherits verbose flag
• too imperative, +4 config options
• not applied to spawned processes outputs, they just go as is to the root process stream

[antongolub]

@JohnnyMarnell, @ryami333,

Let's discuss if it covers your issues.

[antonmedv]

What about removing log from fetch at all? This will be more “browser” like.

[antonmedv]

4 additional API endpoints) I’ll try to do my version of addressing same issues today.

[antongolub]

browser

zx is suitable for crawling too, so fetch/http logging might be useful.

4 additional API endpoints)

I can assemble them into $.log.*. But that will be cheating)

I’ll try to do my version of addressing same issues today.

It's essential to bind log profile to async ctx:

$.profile({logFormat: maskingLogger}, async() => {
  await $.fetch('https://exampe.com', {authorization: ...})
  // fetch https://exampe.com {"Authorization": "23e334ds******23"}
  $`qctl login ${token}`
  // $ qctl login fr324d324******09 
})

$.profile({logIgnore: 'fetch'}, () => {
   // push smth optinal that does not requre logging or even waiting
  for(let url of urls) {
     $.fetch(url, {data: ...})
   }
})

[antonmedv]

zx is suitable for crawling too, so fetch/http logging might be useful.

True.

I’m thinking about something like this:

• true (dedault) - logs everything, to stderr
• cmd only (fetch can be treated as cmd) - logs only commands, no outputs
• false - no logging.

[antongolub]

default stderr

+ zx logs are more of debug/tracing which usually goes to stderr (console.trace)

[antonmedv]

Also echo can also be to stderr. WDYT?

[antongolub]

cmd only (fetch can be treated as cmd) - logs only commands, no outputs

Let's make it configurable like in debug:

log('cmd:input', ...)
log('cmd:output', ...)

$.logIgnore = ['cmd:*'] // to disable all
$.logIgnore = ['cmd:output'] // to disable stdout pushing

External plugins may bring their own log events, so we need to apply filters to them too.

[antongolub]

Also echo can also be to stderr

This might be surprising a bit.

UPD But if logOutput set to stderr this will happen )

[antonmedv]

Lets not allow plugins to use zx logs. I’d like to keep logs of zx as internal as possible.

[antongolub]

Logger is essential.

1. We can introduce protected scope zx and restrict plugins from using it: zx:cmd:input, zx:fetch:input, etc.
2. It's immutable, I don't see any risks

Why log format is needed: we have a special magic process on our kuber pods that captures stdstreams and pushes their contents to elastic index. But it happens only for json strings with special fields, so we have to attach them somehow.

[antongolub]

@antonmedv

Show your concept. I'll try to mimic. 0/1/2 level schema as you mentioned is fine, but it has some limitations. For large scripts level 1 may be not enough for debugging when option 0 produces too big output to recognize.

Instead, I suggest introducing a composite dynamic scope prefix: zx:cmd:input:${cmd} so we could even filter only necessary commands. Maximum granularity

[antongolub]

Idea! We can use standard env.DEBUG value as logIgnore default if specified.

[antonmedv]

zx:cmd:input:${cmd}

too many abstraction. Lets try to find simplest solution. Let’s reduce complexity. Can you expla little bit more your needs?

[antongolub]

too many abstraction

Too many java devs around ))) I'm trying to attach meta to every log event to be able to: 1. process only necessary, 2. define event projection — format and channel

How I'm going to use these powers — wrapping some code blocks with retries and raising log verbosity for the last attempt and every next matching bin.

[antongolub]

Maybe it's too much. Reverted.

[JohnnyMarnell]

My biggest complaint was the nonstandard usage of stdout for diagnostic logging, rendering any shell script looking to provide an "output" problematic. Seeing that's finally being defaulted to stderr recently, great. Beyond that, the logging mechanism doesn't need to be this complex, a simple -v vs -vvvvv verbosity level mechanism would have sufficed. But main thing is if it has sane defaults in addition to stderr, which I think this does...

[antongolub]

Thanks for the feedback. I've tried to combine both approaches:

• verbosity level describes "importance"
• scope marks the event source (fetch, cmd, etc)

[antonmedv]

The v7 will introduce $.log. Closing in flavor of.