Epic: Multiple file output for code generators

[RicoSuter]

Implemented in NJS, update NSwag generators, CLI, UI, etc.

[vgb1993]

Hy Rico, what's the status of this epic?

In my company we are generating an Angular client from our Core API definition. Because all the services are generated inside the same file we can not code-split each service in a separate bundle. This is very inconvenient as you can imagine.

We would like to have each service in it's own file.

I'm not familiar with the Liquid template notation, but I've read in this issue that you could write to multiple files. Do you think this could be achived this way?

Thanks for your time

[ttma1046]

@vgb1993
My approach is

1. put the auto-generated giant service and just itself in one single NgModule. (e.g. api-autogenerated-module).
2. ask different service/facade in different ngModules uses/DI that gaint service in api-autogenerated-module.

[daiplusplus]

I might also suggest running a sed (or T4 script if you're on Windows and have msbuild available) that takes the monolithic output and splits it up into multiple output files. This can be done with a simple regex.

[vgb1993]

@Jehoel

I might also suggest running a sed (or T4 script if you're on Windows and have msbuild available) that takes the monolithic output and splits it up into multiple output files. This can be done with a simple regex.

Hey thanks for the reply. I already read about this option somewhere in this repo, but I was a bit skeptical. Do you have any examples? Also I have two questions about this approach:

• What happens if an interface is shared by two Services in different files? Will the interfaces be declared all in one file or duplicated in multiple files.

• In case all the interfaces where in the same file (wich would make sense to me) how would you generate the imports to the interfaces the Service uses? Maybe something like Import * from "interfaces.ts" would do the job!

Anyway I realy feel this feature would be much better implemented inside NswagStudio, don't you think? We like using the GUI, it's very user friendly. Is there any plan to implement the feature? This has been open since last year.

[vgb1993]

@ttma1046

1. put the auto-generated giant service and just itself in one single NgModule. (e.g. api-autogenerated-module).
2. ask different service/facade in different ngModules uses/DI that gaint service in api-autogenerated-module.

I don't like the idea of writing the Service twice (autogenerated / facade) only to allow code splitting, this defeats the purpose of the generator.

Also, are you sure this allows code splitting? I did some tests some days ago and I beliebe if the services are in the same .ts file they always get bundled together.

Cheers

[RicoSuter]

Internally the feature is almost complete. It's just not exposed via CLI, here you can see that internally we already have multiple "files" (artifacts) which are then merged and written to a single file in the CLI

https://github.com/RicoSuter/NSwag/blob/master/src/NSwag.CodeGeneration/ClientGeneratorBase.cs#L89

[vgb1993]

Hy @RicoSuter,

Yes, I can see the code artifacts. Good to see it's in process. Could you explain us what is left to implement and an approximate deadline? Also I wonder what would the final result look like?

Thanks for the reply and all the great work, NSwag is amazing. Let me know if if you need help with this, I could give you a hand.

[joan-grau]

I'm currently traped in quite the same situation, I've been using NSwag for a while and this feature sounds like a very usefull and convinient one.

Is there any way to access this splited-files generator? Do you have plans to releaseing or exposing this feature via CLI?

Also, If there is some help needed, do not hesitate on contact me, It would be a pleasure to contribute to this amazing project.

[francisminu]

@RicoSuter Can you please let us know when this is planned to be released? Also, if I need to use it now, is it possible?

[rjs-picturepark]

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

[hemiaoio]

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference
nswag-ts-splitter

[emisand]

@RicoSuter any update on this epic?
We have an API that results in a 4 MB nswag typescript file and such a large file is not usable for our application.
The solution is to have one typescript file for each controller.

[Mike-Becatti]

@RicoSuter any update on this epic?
We have an API that results in a 4 MB nswag typescript file and such a large file is not usable for our application.
The solution is to have one typescript file for each controller.

@emisand - We could no longer wait on this badly needed feature . We ended up adding a swagger doc per controller. Then we have an nswag file for each swagger doc. You may have to go this route too.

[JDelladecimas]

I too would like to request having separate files for each model, service and an index.ts file that exports all. @RicoSuter Can you point me to the file that generates this one big typescript file and I could take a look at contributing.

[dariooo512]

Any update on this? Is there any way I can help speed up the release of this feature?

[dylanvdmerwe]

Having a single .ts file get bundled as one really does bloat things.

This is with 11 services:

Ultimately we want to lazy load the services (via modules) and include them as necessary so they are lazy loaded and not all bundles in the main bundle.

[nobba75]

Bump, any update on this @RicoSuter? Thanks for your effort!

[Ldoppea]

Hi,

I'm also interested by this feature.

But I'm a bit lost about what exists or not:

• @RicoSuter talked about some work in progress, but is it available yet? Epic: Multiple file output for code generators #1398 (comment)
• Some issues seem to be about spliting the API documentation (so not what I need) : Split up the generated OpenApi file into multiple files #1905
• @hemiaoio's solution seems to be running on API side and generate a TS file, is that right? Epic: Multiple file output for code generators #1398 (comment)

My need is to split the resulting TS file when using TypeScriptClientGenerator as 1 file is too big and it breaks the Storybook.js' build. I give more details here : https://stackoverflow.com/questions/67116837/is-it-possible-to-make-storybook-js-working-with-very-large-auto-generated-types

Is there some solution existing for this scenario?

If not, is there anything I can do to help? I don't know how this project is implemented nor its philosophy but I can try if you give me some directions.

[Christian-Oleson]

I'll also chime in as this being quite important. Essentially, I pretty much am going to avoid using NSwagStudio in favor of other Open API Code generators that have this feature, which requires more work on my end to customize/extend, but in the long term will allow me to on modify the explicit files that are being changed.

[bsell93]

#1398 (comment)

I've had this same problem. I ended up landing on a solution. Although not desirable it has proven to be effective.

Essentially what I ended up with is using Regex to parse the output and separate it into appropriate files.

Hopefully this helps someone else. I know it's helped my team tremendously in code reviews 😂

Note: This is a .netcore backend with typescript/axios generated client.

GenerateClientFilesAsync (click to expand)

private async Task GenerateClientFilesAsync()
{
    var document = await OpenApiDocument.FromUrlAsync("http://localhost:5000/swagger/v1/swagger.json");
    var settings = new TypeScriptClientGeneratorSettings
    {
        ClassName = "{controller}Api",
        ClientBaseClass = "ClientBase",
        Template = TypeScriptTemplate.Axios,
        UseGetBaseUrlMethod = true,
        UseTransformOptionsMethod = true,
        UseTransformResultMethod = true,
    };
    settings.TypeScriptGeneratorSettings.ExtensionCode = _clientBaseImport;
    settings.TypeScriptGeneratorSettings.DateTimeType = TypeScriptDateTimeType.String;

    var generator = new TypeScriptClientGenerator(document, settings);
    var code = generator.GenerateFile();

    code = RemoveBrokenGeneratedCode(code);
    var (newCodeWithoutEnums, enumNames) = RemoveEnumsAndGenerateClientEnumFile(code);
    code = newCodeWithoutEnums;
    var (newCodeWithoutModels, modelAndInterfaceNames) = RemoveModelsAndGenerateClientModelFile(code, enumNames);
    code = newCodeWithoutModels;
    GenerateClientApiFiles(code, modelAndInterfaceNames, enumNames);
}

RemoveBrokenGeneratedCode (you may or may not need these... I did not, so I removed them) (click to expand)

private string RemoveBrokenGeneratedCode(string code)
{
    code = Regex.Replace(code, @"^(((?!protected).)*)Promise<(.*)>", "$1Promise<AxiosResponse<$3>>", RegexOptions.Multiline);
    code = Regex.Replace(code, @"\s*private instance: AxiosInstance;", "");
    code = Regex.Replace(code, @"\s*this\.instance = instance \? instance : axios\.create\(\);", "");
    code = code.Replace("constructor(baseUrl?: string, instance?: AxiosInstance)", "constructor(baseUrl?: string)");

    code = Regex.Replace(code, @"(else if \(status !== 200 && status !== 204\) {\s*)const _responseText.*;\s*return.*;", "$1return response.data;");

    return code;
}

RemoveEnumsAndGenerateClientEnumFile (click to expand)

private (string code, IEnumerable<string> enumNames) RemoveEnumsAndGenerateClientEnumFile(string code)
{
    var matches = Regex.Matches(code, @"(^export enum\s(.*)\s\{[^\}]+\})", RegexOptions.Multiline);
    var enumNames = matches.Select(m => m.Groups[2].Value).OrderBy(x => x);
    var enumString = String.Join('\n', new string[] { "/* tslint:disable */", "/* eslint-disable */" }.Concat(matches.Select(m => m.Value)));
    foreach (Match match in matches)
    {
        code = code.Replace(match.Value, "");
    }

    WriteOutGeneratedFile("enums.ts", enumString);

    return (code, enumNames);
}

RemoveModelsAndGenerateClientModelFile (click to expand)

private (string code, IEnumerable<string> modelAndInterfaceNames) RemoveModelsAndGenerateClientModelFile(string code, IEnumerable<string> enumNames)
{
    var interfaceMatches = Regex.Matches(code, @"(^export interface (\S+) [\S\s]*?^})", RegexOptions.Multiline);
    var modelMatches = Regex.Matches(code, @"(^export class (\S+) [^\n]*implements[\S\s]*?^})", RegexOptions.Multiline);
    var enumNamesString = String.Join(", ", enumNames);
    var importEnumString = $"import {{ {enumNamesString} }} from './enums';";
    var modelString = String.Join('\n',
        new string[] { "/* tslint:disable */", "/* eslint-disable */", importEnumString }
            .Concat(interfaceMatches.Select(m => m.Value))
            .Concat(modelMatches.Select(m => m.Value)));
    foreach (Match match in interfaceMatches)
    {
        code = code.Replace(match.Value, "");
    }
    foreach (Match match in modelMatches)
    {
        code = code.Replace(match.Value, "");
    }

    var modelAndInterfaceNames = modelMatches
        .Select(m => m.Groups[2].Value)
        .Concat(interfaceMatches.Select(m => m.Groups[2].Value))
        .OrderBy(x => x);

    WriteOutGeneratedFile("models.ts", modelString);

    return (code, modelAndInterfaceNames);
}

GenerateClientApiFiles (click to expand)

private void GenerateClientApiFiles(string code, IEnumerable<string> modelsAndInterfacesNames, IEnumerable<string> enumNames)
{
    var isAxiosErrorString = code.Split('\n').TakeLast(4);
    var apiMatches = Regex.Matches(code, @"(^export class (\S+) [^\n]*extends ClientBase[\S\s]*?^})", RegexOptions.Multiline);
    foreach (Match match in apiMatches)
    {
        string apiCode = match.Value;
        var enumsToImport = enumNames.Where(x => Regex.IsMatch(apiCode, $@"\b{x}\b"));
        var modelsToImport = modelsAndInterfacesNames.Where(x => Regex.IsMatch(apiCode, $@"\b{x}\b"));
        apiCode = apiCode.Replace("export class", "export default class");

        var numberCommentAndImportLines = 11;
        if (modelsToImport.Any()) ++numberCommentAndImportLines;
        var fileTopImportsAndComments = code.Split('\n').Take(numberCommentAndImportLines);
        var apiString = string.Join('\n', fileTopImportsAndComments.Concat(new string[] { apiCode }).Concat(isAxiosErrorString));

        if (enumsToImport.Any())
        {
            var enumNamesString = String.Join(", ", enumsToImport);
            var importEnumNamesString = $"import {{ {enumNamesString} }} from './enums';";
            apiString = apiString.Replace(_clientBaseImport, $"{_clientBaseImport}\n{importEnumNamesString}");
        }

        if (modelsToImport.Any())
        {
            var modelAndInterfaceNamesString = String.Join(", ", modelsToImport);
            var importModelAndInterfaceNamesString = $"import {{ {modelAndInterfaceNamesString} }} from './models';";
            apiString = apiString.Replace(_clientBaseImport, $"{_clientBaseImport}\n{importModelAndInterfaceNamesString}");
        }

        WriteOutGeneratedFile($"{match.Groups[2].Value}.ts", apiString);
    }
}

The output of this looks something like this in my folder/file structure;

|generated
|-- enums.ts
|-- FilesApi.ts
|-- models.ts
|-- OrganizationsApi.ts
|-- UsersApi.ts

You get the gist

[daiplusplus]

I have already implemented multiple-file-output in my local clone last year...

The problem is I wrote it for myself - using my own opinionated coding-style and well... it's far too different than @RicoSuter's style for them to want to simply accept a PR - so I'd have to spend a decent amount of time reworking it into the house-style through gritted teeth...

UPDATE: I don't just mean stuff that can be automated via .editorconfig + Reformat, but more fundamental stuff... actually, come to think about it, it isn't that bad... I'll take a look at my code again and get back to this thread shortly...

@RicoSuter I assume you'll be okay if my contributions make changes to the NSwag Studio UI and add more code-gen options and include my customized NSwag Liquid Templates as in-box alternatives for other people to use? I really like how I've gotten them to work and I'd like to share them - so if you include my new features in the product as an ego-stroking exercise for myself I'll get my multi-file-output support ready for PR 😸

[daiplusplus]

@bsell93 I just noticed you have HTML5 <summary>-expanding regions in your post, I didn't realise GitHub's Markdown supported that - so I thought your post was just daydreaming, not providing an actual solution... you might want to make it more obvious that your post does contain code.

[FaizulHussain]

Badly needing this feature to be OOB, file is getting huge here :(

[llgarrido]

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference
nswag-ts-splitter

@hemiaoio I liked your quick solution as long as this issue does not have a definitive solution.

There are aspects that you have not yet addressed.

• You haven't taken into account the imports required for specific framework templates such as Angular.
• You didn't identify classes from TypeScript union types, e.g. (string | number | undefined).
• Provide the possibility to create the files with names in kebab-style.

I want to publish a fork of your great work with my suggestions.

[llgarrido]

I have already implemented multiple-file-output in my local clone last year...

The problem is I wrote it for myself - using my own opinionated coding-style and well... it's far too different than @RicoSuter's style for them to want to simply accept a PR - so I'd have to spend a decent amount of time reworking it into the house-style through gritted teeth...

UPDATE: I don't just mean stuff that can be automated via .editorconfig + Reformat, but more fundamental stuff... actually, come to think about it, it isn't that bad... I'll take a look at my code again and get back to this thread shortly...

@RicoSuter I assume you'll be okay if my contributions make changes to the NSwag Studio UI and add more code-gen options and include my customized NSwag Liquid Templates as in-box alternatives for other people to use? I really like how I've gotten them to work and I'd like to share them - so if you include my new features in the product as an ego-stroking exercise for myself I'll get my multi-file-output support ready for PR 😸

How did you manage to split the files? From Liquid templates or C#?

Is this implementation in your github fork? Where is it?

Thanks

[daiplusplus]

@llgarrido

How did you manage to split the files? From Liquid templates or C#?

In C#

Is this implementation in your github fork? Where is it?

I wrote "my local clone" not "my fork" - the code exists only on my PC. I haven't pushed it anywhere.

[Mike-Becatti]

@Jehoel - I did it in C# by creating a document per controller.

   var manager = (ApplicationPartManager)services.LastOrDefault(d => d.ServiceType == typeof(ApplicationPartManager)).ImplementationInstance;
   var feature = new ControllerFeature();
   manager.PopulateFeature(feature);

   // Get controllers, but exclude BaseController class from generating a swagger document.
   List<string> controllerNames = feature.Controllers.Where(c => !c.Name.Contains("BaseController")).Select(t => t.Name).ToList();

   controllerNames.ForEach(c =>
   {
      string controllerName = c.Replace("Controller", string.Empty);
      services.AddSwaggerDocument(settings => { settings.ApiGroupNames = new string[] { controllerName }; settings.DocumentName = controllerName; });
   });

[daiplusplus]

@AlphaCreative-Mike That's horrible :S - you shouldn't compromise the design of a system to workaround minor technical or workflow issues.

[Mike-Becatti]

@AlphaCreative-Mike That's horrible :S - you shouldn't compromise the design of a system to workaround minor technical or workflow issues.

We own both sides of the system. Sometimes doing what is practical overrides what should be done in principle. To each their own. What's your alternative? This issue has been open for 3 years. Are you going to keep waiting for the fix?

[daiplusplus]

@AlphaCreative-Mike

What's your alternative?

Years ago, I cloned NSwag locally, modified its source to generate multiple output files - and numerous other changes to suit my own opinionated style and architecture.

Are you going to keep waiting for the fix?

I already have my own fix. And I'd love to share it, but there are three problems:

• My changes have evolved with me and my local clone is now very, very diverged from the current canon repo - not just the C# code, but the Liquid templates have been mutilated beyond recognition, and the Liquid templates and my C# changes are also very tightly-coupled together.
• Even if rebasing+retro-updating weren't an issue my changes were written according to my opinionated style and architecture ("my own way of doing things") which will very likely not be accepted by @RicoSuter as is
• Finally, @RicoSuter still hasn't yet replied to say if he'd be interested in having my changes at all - or if he's still planning on using his original approach he aluded to here: Epic: Multiple file output for code generators #1398 (comment) - until I get approval from him there's no point to me spending significant portion of my time to reimplement my old work on the current project HEAD.

[Mike-Becatti]

@Jehoel - That's awesome. If your changes get incorporated into this repo I'll pick them up and toss my 'solution' in the trash. Until then, my solution suits my needs.

[lucaritossa]

Hi everyone!
I'm working in a project based on ASP.NET WebAPI and Angular v10 library.
The project is growing every day and a few days ago I crashed my head to a strange issue building angular library: "Maximum call stack size exceeded"

I finally found that the problem is generated by the size of "api.generated.ts" (this is the name of the output file of NSwag generation). Not a very big file size, it's about 1.6MB and it's still a mystery for me why it happens.

For now I found a work-around forking NSwag, changing/adding some lines of ClientGeneratorBase and TypeScriptFileTemplateModel (see lucaritossa@872ead1), publishing a custom version of NSwag.MSBuild in our private nuget repository and using it in our solution. Some changes in nswag.config and angular File.liquid template complete the work-around.

Splitting api.generated.ts into 2 files

• api-client.generated.ts
• api-dto.generated.ts

is possible having 2 nswag config file into the same .NET project.
I prepared

• nswag-client.config with "generateClientClasses": true and "generateDtoTypes": false
• nswag-dto.config with "generateClientClasses": false and "generateDtoTypes": true

With the split, a problem must be solved:
api-client.generated.ts does not compile because missing import of dto classes, now defined to the other file.

To solve the problem I needed to instruct typescript File.liquid template of api-client to write down the "import { dto classes } from './api-dto.generated"

Here it is the snippet

{%-        if Framework.IsAngular -%}

{%-            if Framework.UseRxJs5 -%}
...
...
{%-            endif -%}

{%-            if GenerateDtoTypes == false -%}
import { 
{% for name in TypeNames %}
  {{ name }},{% endfor -%}

} from './api-dto.generated';
{%-            endif -%}

{%-        endif -%}

GenerateDtoTypes and TypeNames are 2 properties I forcibly added to TypeScriptFileTemplateModel

Finally, to avoid a massive refactoring of the import currently defined in hundreds of angular services/components based on 'api.generated' I defined this file with the export of the other two

export * from './api-dto.generated';
export * from './api-client.generated';

THIS IS A WORKAROUND, IS NOT the solution of this epic!
Defining 2 nswag config slows down the compilation (precious seconds) and it is required a little change to liquid template (this disturbing me because more attention will be required when updating NSwag packages)

BUT, I ask to @RicoSuter if I can propose a PR of the changes lucaritossa@872ead1 to avoid the custom version of NSwag.MSBuild I have currently in my private nuget repo and to help others in case they want to replicate my workaround.

[nkosi23]

What is the status of this feature? Is help needed for anything to get it ready?

Our use case is that we do not want "confidential" services that are only used by the staff portal to be present in the client shipped to the general public. While the backend has proper security mechanisms to prevent unauthorized access, we do not want people taking a look at the source code to be able to know our business processes (function names, class names, DTO & Cie leak a lot of information).

[williamleeadc]

It's already July 5th, 2024, and I still haven't seen nswag being able to modularize the generation of front-end proxy classes!

[patrickklaeren]

What is the status of this feature? Is help needed for anything to get it ready?

Our use case is that we do not want "confidential" services that are only used by the staff portal to be present in the client shipped to the general public. While the backend has proper security mechanisms to prevent unauthorized access, we do not want people taking a look at the source code to be able to know our business processes (function names, class names, DTO & Cie leak a lot of information).

You can generate different outputs for your client and confidential client.

[nkosi23]

@patrickklaeren Thanks for the feedback, I'll take a look!
Do you have any top of the mind pointers in the doc?

[hemiaoio]

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference
nswag-ts-splitter

@hemiaoio I liked your quick solution as long as this issue does not have a definitive solution.

There are aspects that you have not yet addressed.

• You haven't taken into account the imports required for specific framework templates such as Angular.
• You didn't identify classes from TypeScript union types, e.g. (string | number | undefined).
• Provide the possibility to create the files with names in kebab-style.

I want to publish a fork of your great work with my suggestions.

Absolutely, I'm eagerly anticipating your piece.