TypeDoc JavaScript heap out of memory 0.18.0

[nedkelly]

I've been having trouble with TypeDoc and up until yesterday I was under the impression that TypeDoc was at fault, but after a lot of digging I reduced my failing docs build down to a small subset of components, they all had one thing in common, schema-dts. I posted on an existing open issue with them to get their input, they suggested I also open an issue here as there is still an expectation that TypeDoc should be able to traverse large type trees without leading to OOM errors and frankly I have to agree with that logic.

I created a minimal repo to test against and found that I could easily reproduce the issue. My project was using schema-dts@0.4.4 and I was getting a JavaScript heap out of memory error, upgrading to schema-dts@0.6.0 fixed a number of my files but still hangs on some. The schema-dts@0.5.0 and up releases have a number of improvements specifically relating to improving TypeScript performance. Setting --max_old_space_size=8048 prevents the OOM error but the build hangs indefinitely.

The schema-dts types are massive, I can't say if they should be done differently to limit the overhead placed on the compiler but we require these type definitions in order to adhere to schema.org structured data throughout our systems.

TypeDoc: 0.18.0 (also rolled all the way back to 0.15.0 and had similar results)
TS: 3.9.7
Node: v12.18.0
Ubuntu 20.04 (Windows 10 WSL2)

Related schema-dts issues:
google/schema-dts#34

Minimal Example:
https://github.com/nedkelly/typedocissue

Any help would be much appreciated.

[Gerrit0]

Thank you for the detailed reproduction! I spent a bit of time looking into this, but wasn't able to quickly identify a fix... and I'm not sure an easy fix exists.

TypeDoc currently has a few major problems with large projects, the most major being in the type resolution stage. Most of TypeDoc was written when namespace modules (as in module Foo { }) was common, and as a result of that, the method it currently uses to figure out what a reference type points to is performing a search up through declarations until it finds one with the same name. This, eventually, results in searching every single item documented... However, in this case there is another issue.

Types are incredibly difficult to deal with.... by which I mean they are next to impossible to figure out how to display... unless you have a type node. The return type of getSubjects isn't annotated (casts don't count, they don't produce a type node). You can get an inkling of what TypeDoc tries to convert when you hover over getSubjects.

function getSubjects(articleContent: Article): ((string & {
    name: string;
    disambiguatingDescription: string;
}) | ({
    "@type": "Article";
} & ThingBase & {
    about?: string | ({
        "@type": "Article";
    } & ThingBase & ... & {
        ...;
    }) | ... 805 more ... | Thing[];                                           <----
    ... 95 more ...;
    workTranslation?: ({
        ...;
    } & ... 2 more ... & {
        ...;
    }) | ... 152 more ... | CreativeWork[];
} & {
    ...;
} & {
    ...;
}) | ... 804 more ... | ({                                                         <-----
    ...;
} & ... 2 more ... & {
    ...;
}))[]

Trying to annotate the type also fails, this is definitely a bug.

export default function getSubjects(
  articleContent: Article
): (Thing & {
  name: string;
  disambiguatingDescription: string;
})[] {

The cause of this is that TypeDoc doesn't currently have a parenthesized type node converter... so falls back on the generic type converter which doesn't use a node and falls straight back into the trap. Once annotated with Array<> instead of parenthesis, everything works perfectly.

export default function getSubjects(
  articleContent: Article
): Array<Thing & {
  name: string;
  disambiguatingDescription: string;
}>[] {

This is still a problem. I can't help but feel something majorly wrong is going on when trying to convert without the type, but it at least provides a workaround for now...

[nedkelly]

Wow @Gerrit0 this is an excellent explanation, thanks for taking the time to respond to this issue. It's certainly a difficult problem to solve and I'm extremely appreciative of the efforts of the contributors of this project. I'll give your workaround a try, at the end of the day I agree that there are bugs to be dealt with but if adding some extra annotations helps in the meantime it's not so bad because it's still valid syntax.

Thanks again.

[Gerrit0]

Having looked at this a bit more, I think TypeDoc is doing the right thing, it is just that the resolution stage is so incredibly inefficient we end up running out of memory. I've just added a parenthesized type node converter, which fixes the fixable part of this issue.