New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

What will Domain be replaced with? #10843

Open
isaacs opened this Issue Jan 17, 2017 · 27 comments

Comments

Projects
None yet
@isaacs
Contributor

isaacs commented Jan 17, 2017

I'm adding support for parallel tests to node-tap. tapjs/node-tap#306

One sticky point is that, like most test frameworks, node-tap needs to be able to cleverly handle thrown errors and tie them to the test environment where the error was thrown from (especially since many existing assertion libs turn things like expect(foo).to.match(/regexp/) into a thrown error).

When tests were not run in parallel, there was no problem. Just walk down the tree of test.currentChild or whatever, and a single process.on('uncaughtException') is sufficient.

I'm using Domains to manage thrown errors in child tests, and it works great, but the bright red deprecation notice in the docs is scary.

What will it be replaced with? Will it be feature-equivalent? If so, is there any reason why the existing API has to be broken, and not re-implemented in terms of the new thing? Is there an expected time frame for the Domain api to be broken?

Thanks.

@arciisine

This comment has been minimized.

arciisine commented Jan 17, 2017

While not official, something like async-listener or zonejs should generally be able to fit the bill, with some caveats.

@jasnell

This comment has been minimized.

Member

jasnell commented Jan 17, 2017

At this point it's not clear at all what will replace it. There is this zones proposal being discussed at the TC-39 level but it's not something that is far enough along for us to be able to make any reasonable decisions on. At the same time, if there is going to be something happening at the TC-39 level, it doesn't make much sense for us to come up with a likely incompatible replacement. So at this point, the deprecation/replacement of domain is in a holding pattern and will be for the foreseeable future.

Any discussion on a potential replacement needs to happen in the node-eps repository.

@addaleax

This comment has been minimized.

Member

addaleax commented Jan 17, 2017

My understanding was that async_hooks, when it’s ready, would be suitable for having domain re-implemented on top of it, with a similar or identical API?

/cc @nodejs/diagnostics

@jasnell

This comment has been minimized.

Member

jasnell commented Jan 17, 2017

That's a possibility but still runs up against the possibility of being inconsistent/incompatible with whatever language level construct is settled on by TC-39. It won't do us much good to put the effort into a domains v2 if we end up having to deprecate it in favor of a language level thing, etc

@sam-github

This comment has been minimized.

Member

sam-github commented Jan 17, 2017

Is there an expected time frame for the Domain api to be broken?

@isaacs I was under the impression that it is broken already. Besides https://github.com/nodejs/node/blob/master/doc/topics/domain-postmortem.md, promise users report problems with domains, and users of async/await have found them broken, too, IIRC.

If you don't use any of the broken bits, they seem to work fine, but its a bit of a mine field out there.

@Qard

This comment has been minimized.

Member

Qard commented Jan 17, 2017

It's really been "broken" to some degree pretty much since day one. It just gets more and more broken as new language/runtime features are introduced and the necessary parts to support domains don't stay in sync. It never quite satisfied the needs and really just exposed a huge hole where people wanted features. That's a big part of why I pushed to start the diagnostics working group in the first place--to try to better deliver what was needed. The new 'async_wrap' stuff is definitely a step in the right direction, but I personally feel there's still a long way to go. I hope the zones effort grows into something more solid. TC39 stuff takes time though. 😕

@isaacs

This comment has been minimized.

Contributor

isaacs commented Jan 18, 2017

@Qard @sam-github It's not "broken already", it is what it is. I am using "broken" in a very technical non-normative sense here, meaning "change the API in a way that requires changes to the code using it".

You might not like the Domains API that exists today, and I'd probably agree with your criticisms of it. But it does a specific job very reliably, which I can't do any other way.

My fear is that, after spending 2 years in a "deprecated" state, someone will decide it's time to rip it out without giving me time to upgrade (or worse, without there even being anything to upgrade to). Could the documentation at least provide some clarity as to the requirements that have to be met before Domains will be removed? Or better yet, could we just deprecate the parts that are known to be harmful? (d.enter(), for example, should probably never be used in userland code, and it was a mistake to expose it.)

@jasnell

This comment has been minimized.

Member

jasnell commented Jan 18, 2017

There are no plans, nor has anyone suggested, nor would I expect anyone on @nodejs/ctc to suggest or even reasonably consider removing domain without a clear replacement in place. It is likely to be left alone as is without any modifications for the foreseeable future.

@Fishrock123

This comment has been minimized.

Member

Fishrock123 commented Jan 18, 2017

I'll try to read though this tomorrow, but the idea was to get async-hooks in in a state that would allow a user module to correctly replicate domains (and other variations of the async-error-handling idea), and then move towards removing it.

The resulting module would probably be more maintainable (perhaps even actually maintained!), and core's APIs for it a lot more platform-like and unoppinionated.

There are no plans, nor has anyone suggested, nor would I expect anyone on @nodejs/ctc to suggest or even reasonably consider removing domain without a clear replacement in place.

I would like to be clear that "the replacement" may not have the same level of API or do the same thing, but it will certainly be pushed towards allowing replication of the domain API as much as/if possible.

@Qard

This comment has been minimized.

Member

Qard commented Jan 18, 2017

@isaacs I hear you on that, and I don't expect it'll just disappear immediately when something new comes along. It's possible the existing API might even just get ported to async-hooks in the future. It's really just the internal behaviour that's problematic: lack of support for promises and asyn/await, inconsistent timings between node versions, etc. My feeling is that the current deprecation is really more about it being potentially unreliable than there being any specific intention to fully replace it currently.

@ORESoftware

This comment has been minimized.

Contributor

ORESoftware commented Mar 29, 2017

I am in the same exact boat as @isaacs; every test runner in the Node.js ecosystem that is trying to parallelize tests will have the same exact problem - the problem being: the only reliable way to trap errors thrown in an arbitrary chunk of code* is domains.

As for native promises being broken with domains, you can try this:

const then = Promise.prototype.then;
Promise.prototype.then = function (fn1, fn2) {

  if (process.domain) {
    fn1 = fn1 && process.domain.bind(fn1);
    fn2 = fn2 && process.domain.bind(fn2);
  }
  return then.call(this, fn1, fn2);
};

Not super well-tested, just throwing that code out there.

Bluebird promises have had support for domains for more than 2 years TMK, so that's been a good thing for domain users for awhile.

*Where that code is user-provided or otherwise out of your control.

@syrnick

This comment has been minimized.

syrnick commented Jul 14, 2017

Note on promises. Native v8 promises don't respect domains - the callbacks aren't called in the same domain. Polyfills work fine, because they are just library level constructs. Hence, domains mostly work with transpiled async/await and with most libraries that use promises if the global Promise is the polyfill, not the native Promise.

It's obviously a confusing state and using domains has all sorts of caveats and gotchas. Still, immensely useful creature.

@misterdjules

This comment has been minimized.

Contributor

misterdjules commented Jul 14, 2017

@syrnick

Native v8 promises don't respect domains

Does #12489 solve this problem for your use case? (note: it's only available in node 8.x).

@syrnick

This comment has been minimized.

syrnick commented Jul 14, 2017

nice! I'll test that out on 8.x.

@Trott

This comment has been minimized.

Member

Trott commented Aug 10, 2017

domain does have that funny "pending deprecation" notice. Now that async-hooks has arrived, are we anywhere close to actually giving domain a doc-only deprecation that doesn't have the "pending deprecation" ambiguity/confusion? Or not yet? I have no interest in pushing for a runtime deprecation or anything like that, but I would like the docs to be clear about the status.

@BridgeAR

This comment has been minimized.

Member

BridgeAR commented Sep 23, 2017

I think that is a good idea @Trott

@owenallenaz

This comment has been minimized.

owenallenaz commented May 31, 2018

As I've been utilizing async and await, it seems apparent to me that with a proper execution of Promise chains, the domain module no longer seems necessary. Steps across the event loop can be maintained via the Promise chain, and thus errors can properly be try/catch no longer how deep and handled accordingly. Given correct implementation by Module implementors, which will take some time, the domain module no longer seems necessary. So far from my experimentation, I'm not aware of an edge case that I have used domains for which Promise chains do not handle and async and await provide the needed syntactic sugar to make it reasonable to utilize.

So then with that in mind, is the replacement, simply start using Promises.

@misterdjules

This comment has been minimized.

Contributor

misterdjules commented Aug 21, 2018

@owenallenaz Just to make sure I understand what you suggest: do you mean that errors thrown from an async task not originally scheduled by a promise should be "wrapped" by a promise, such as for instance using the following to handle errors thrown from setTimeout callbacks?

const somePromise = new Promise((resolve, reject) => {
  setTimeout(function () {
    try {
      throw new Error('boom!');
    } catch (e) {
      reject(e)
    }
  })
})

async function mainAsync() {
  try {
    await somePromise;
  } catch (err) {
    console.error(err);
  }
}

mainAsync();
@owenallenaz

This comment has been minimized.

owenallenaz commented Aug 22, 2018

With callback semantics we're currently forced to deal with errors in two ways. The first is via the callback and the second is thrown errors caught by domain. I spun up a simple site in a gist to showcase the problem.

https://gist.github.com/owenallenaz/545bacc327ff1969c56d69e1afb4cfa7

Visit / to see a list of the urls.

  • /callback_valid/ - Valid response
  • /callback_cb_error/ - Error cb'd due to DNS failure.
  • /callback_thrown_error/ - Error thrown by JSON.parse() failure, causes a process crash.
  • /callback_timeout/ - Error thrown instead a timeout, process crash.
  • /promise_valid/ - Valid promise result.
  • /promise_cb_error/ - Error cb'd due to DNS failure.
  • /promise_thrown_error/ - Error thrown by JSON.parse() failure. Does not crash.
  • /promise_timeout/ - Error thrown after a timeout. Does not crash.

In CPS style if the error is cb'd we're fine. If there is a thrown error, nested within the CPS chain there is simply no way to handle that error other than a domain. With Promise and async/await, we can have one form of error handling, try/catch, and it is able to handle the error no matter how it came to be, nor how deeply nested. Looking at the various promise urls helps to showcase that capability. This capability requires the whole nested chain to be properly executed. But if module developers ensure that their calls are properly handling promise chains, then they should bubble up until someone applys a then in traditional syntax or a try/catch via await syntax.

@Trott

This comment has been minimized.

Member

Trott commented Nov 8, 2018

My impression is that the answer to the "when will domains be removed?" question is "quite possibly never." And the answer to the "what will domains be replaced with?" question is "kinda sorta async_hooks but mostly kinda sorta." While perhaps not the most satisfying answers, I wonder if that's basically what we have, and if so, perhaps this can/should be closed?

@misterdjules

This comment has been minimized.

Contributor

misterdjules commented Nov 13, 2018

@Trott I feel the current state of domain does not send a clear message to users though: on one hand, domain is deprecated, and the documentation hints at "a replacement API", but it doesn't seem like any replacement has been considered.

Some users have been very vocal about the confusion that this message generates, and it seems that we could at least provide user-land alternatives that would allow for a reasonable migration path for users while allowing core to fully deprecate domain.

I've submitted a PR that enables user-land implementations of domain at #24279 (this is a work in progress, but I'd like to gather initial feedback on whether core would be open to merge something similar), and worked on a proof of concept of a user-land domain implementation that relies on those changes at https://github.com/misterdjules/domaine.

Any thoughts on that?

@devsnek

This comment has been minimized.

Member

devsnek commented Nov 13, 2018

lots of replacements have been considered, just nothing has been very compelling (imo because use case itself is uncompelling). I think the best direction would be to suggest that users migrate away from the domain model, and properly handle errors.

@AiDirex

This comment has been minimized.

AiDirex commented Nov 14, 2018

@devsnek how do you provide context for logs of your applications/servers?

@devsnek

This comment has been minimized.

Member

devsnek commented Nov 14, 2018

@AiDirex you could use async_hooks for that. (check out an example here). the thing that has been the problem isn't the scoped context, it's been the error handling.

@AiDirex

This comment has been minimized.

AiDirex commented Nov 14, 2018

@devsnek Thank you, but I know that modules like cls-hooked exist that use async-hooks to solve the issue of context. The problem is that some modules aren't compatible with async-hooks yet and their maintainers won't fix the compatibility issues because async-hooks module is still in an experimental state.

@devsnek

This comment has been minimized.

Member

devsnek commented Nov 14, 2018

@AiDirex out of curiosity, can you point me to a module that is incompatible with async_hooks? every async primitive in node (via js or c++) should be tracking async_hook context.

@AiDirex

This comment has been minimized.

AiDirex commented Nov 14, 2018

@devsnek one is bluebird which lots of other modules depend on it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment