process: add deferTick

[ronag]

Adds a new scheduling primitive to resolve Zalgo when mixing traditional Node async programming (events and callbacks) with the micro task queue (i.e. async/await, Promise, queueMicrotask).

We cannot "fix" nextTick without breaking the whole ecosystem. nextTick usage should be discouraged and we should try to incrementally move to this new primitive.

queueMicrotask is not sufficient to solve this as it is re-entrant.

See following examples:

import { EventEmitter } from 'events'

process.on('uncaughtException', err => {
  console.error("###", err)
})

// Unhandled exception
setImmediate(async () => {
  const e = await new Promise(resolve => {
    const e = new EventEmitter()
    resolve(e)
    process.nextTick(() => {
      e.emit('error', new Error('nextTick'))
    })
  })
  e.on('error', () => {})
})

// Unhandled exception
setImmediate(async () => {
  const e = await new Promise(resolve => {
    const e = new EventEmitter()
    resolve(e)
    queueMicrotask(() => {
      e.emit('error', new Error('queueMicrotask'))
    })
  })
  e.on('error', () => {})
})

// OK, but slow
setImmediate(async () => {
  const e = await new Promise(resolve => {
    const e = new EventEmitter()
    resolve(e)
    setImmediate(() => {
      e.emit('error', new Error('setImmediate'))
    })
  })
  e.on('error', () => {})
})

// OK
setImmediate(async () => {
  const e = await new Promise(resolve => {
    const e = new EventEmitter()
    resolve(e)
    process.deferTick(() => {
      e.emit('error', new Error('deferTick'))
    })
  })
  e.on('error', () => {})
})

Refs: #51156
Refs: #51280
Refs: #51114
Refs: #51070
Refs: nodejs/undici#2497

[benjamingr]

My preference is:

• Educate people better in the docs about attaching error listeners (and .catch listeners) synchronously as the alternative is crashing. Ideally recommend tooling to catch this (?).
• If we want a technological solution, have a mechanism (like with rejections) where we "keep" the error event and re-emit it when an error handler is attached if it's attached before all microticks have run (if no error handler is attached by then we "blow up" like now). This is still risky since it changes "error" timing but less than changing scheduling fundamentally - this is probably fine since it only applies if no "error" handler is attached.

I am -0.5 on adding more scheduling primitives, we can abuse a web standard for this if we really want a new API (like scheduler.postTask with a custom priority but honestly - this stuff is too complicated for our users anyway and I believe increasing the API surface will confuse a lot of them.

[ronag]

This has the same bug just moved right? If someone adds the callback inside a deferTick (caused either by them or the library) they get the same behavior.

No... I'm pretty sure this resolves it properly since it's not re-entrant. Can you provide an example?

I suspect if we encourage people to deferTick we'll get the same bug just later and complicate our scheduling which would ideally be just queueMicrotask and promises (and we should discourage nextTick since there is a parallel standard mechanism).

I think you might be wrong. Again do you have an example)

So I don't think there is a way around "educate people to attach error events synchronously since they are emitted with microtask queue semantics" - this is the case with promises APIs anyway.

What do you mean with promises api anyway? Example?

[ronag]

Please don't get hung up on just solving the error handling case. We have the same problem with any other event. In those cases we have a deadlock instead of an uncaught error, which is even worse.

[ShogunPanda]

LGTM!

[mcollina]

lgtm

[github-actions]

The notable-change PRs with changes that should be highlighted in changelogs. label has been added by @ronag.

Please suggest a text for the release notes if you'd like to include a more detailed summary, then proceed to update the PR description with the text or a link to the notable change suggested text comment. Otherwise, the commit will be placed in the Other Notable Changes section.

[ronag]

I'm putting blocked label on this for now to make sure it doesn't land without more reviews. In particular I would appreciate feedback from @benjamingr and @jasnell

[nodejs-github-bot]

CI: https://ci.nodejs.org/job/node-test-pull-request/56809/

[benjamingr]

This has the same bug just moved right? If someone adds the callback inside a deferTick (caused either by them or the library) they get the same behavior.

No... I'm pretty sure this resolves it properly since it's not re-entrant. Can you provide an example?

I'm not sure what you mean by re-entrant in this context but basically:

// Unhandled exception
setImmediate(async () => {
  const e = await new Promise(resolve => {
    const e = new EventEmitter()
    resolve(e)
    process.deferTick(() => {
      e.emit('error', new Error('queueMicrotask'))
    })
  })
  // also listen in a defer tick callback, presumably because of a library
  process.deferTick(() => {
    e.on('error', () => {})
  });
})

Where the example is contrived (like the others) and in real-world usage the deferTick call would likely come from a library trying to "do the right thing" with a callback

[benjamingr]

What do you mean with promises api anyway? Example?

Just the fact you have to attach an error listener synchronously or you get an unhandled rejection - just like emitter errors

[ronag]

@benjamingr I feel we are out of sync. Do you have any possibility for a 15 min call?

[ronag]

Where the example is contrived (like the others) and in real-world usage the deferTick call would likely come from a library trying to "do the right thing" with a callback

Exactly, this is something that would be mostly used by library developers. We have already the problem with e.g. undici.request which returns a Promise<Stream> and there is no perfect solution atm without process.deferTick.

[benjamingr]

@benjamingr I feel we are out of sync. Do you have any possibility for a 15 min call?

Due to war I'm less available but I'll come to the meeting in 3m but also you have my phone number and are always welcome to whatsapp me.

which returns a Promise and there is no perfect solution atm without process.deferTick.

The question is what happens if the promise resolution is itself deferred by a deferTick in this case for example which would again prevent people from adding an error listener ahead of time.

[jasnell]

I'm not thrilled with the idea of adding Yet Another Scheduling Primitive but not sure there's much of a choice if we want these cases to Just Work... so, LGTM