Create metadata entries generator

[araujogui]

I believe parsers should only convert source files to ASTs with minimal or zero transformation. The current AST to metadata entries transformation should be extracted into a separate generator. Other generators can depend on this metadata generator if needed. This would allow generators to choose between working directly with the raw AST or a more structured metadata representation, depending on their needs.

[avivkeller]

Sounds good to me, however, I don't want to maintain unused code, so is there an actual need for this in our current codebase?

[araujogui]

I believe legacy-json and llms-txt generators can benefit from this. legacy-json for example uses buildHierarchy to transform the flattened entries into a hierarchical structure again.

[araujogui]

I can work on this if it sounds like a good idea for the whole team. @nodejs/web-infra

[flakey5]

+1 to this

The thing for the legacy-json generator though is that it needs the metadata entries. If we switched to using the raw ast for it, it would need to be rewritten almost entirely to use the raw ast, which also means we'd need to do a lot of the stuff in the generator that the metadata parsing is already doing.

The main thing I think that would benefit the generator though (and the new json generator and possibly some others in the future) is some ast-hierarchized generator, where it doesn't pass in flattened entries. The current buildHierarchy impl works off just a hueristic that assumes the order of the entries is the same as in the raw ast (tbf it should be, but there's nothing preventing it from not being like that). But, that's probably for another issue to discuss