Consider spliting Components into separated files

[bartoszirzyk]

Hi,
I've got an openapi.yaml file which has 64k lines. Generator produces Types.swift with 135k lines. It's a big file, hard to navigate and scroll through. It'd be nice to have each Schemas element in separated file and maybe other generated types. Probably it's not what we want as a default behaviour but having a feature flag to enable such thing would be appreciated.

[czechboy0]

Hi @bartoszirzyk - the current behavior is mainly driven by the fact that the recommended way to use the generator is through the build plugin, and the build plugin requires the name of the output files to be known before the OpenAPI document is parsed (due to how SwiftPM plugins work). So unfortunately I don't really see this happening until SwiftPM relaxes that requirement.

Out of curiosity, when do you find you need to look at the generated code, as opposed to looking at the source OpenAPI document and then using autocompletion? The generated code is considered an implementation detail that adopters should have to look at much.

[simonjbeaumont]

FWIW I think this is a reasonable ask. Experimentally, it seems that the compiler is having a much harder time with one large file (cf. more, smaller files). An example on my machine shows pretty substantial improvement:

% wc -l Sources/GithubTypesOneFile/*
457133 Sources/GithubTypesOneFile/Types.swift

% swift package clean && swift package update && swift build --skip-update --target GithubTypesOneFile
Build complete! (647.71s)

% wc -l Sources/GithubTypesMajorSplit/*
 10492 Sources/GithubTypesMajorSplit/APIProtocol.swift
312924 Sources/GithubTypesMajorSplit/Components.swift
133735 Sources/GithubTypesMajorSplit/Operations.swift
    15 Sources/GithubTypesMajorSplit/Servers.swift
457166 total

% swift package clean && swift package update && swift build --skip-update --target GithubTypesMajorSplit
Build complete! (412.02s)

% wc -l Sources/GithubTypesComponentsSplit/*
 10492 Sources/GithubTypesComponentsSplit/APIProtocol.swift
    13 Sources/GithubTypesComponentsSplit/Components.swift
    30 Sources/GithubTypesComponentsSplit/Headers.swift
133735 Sources/GithubTypesComponentsSplit/Operations.swift
   657 Sources/GithubTypesComponentsSplit/Parameters.swift
    15 Sources/GithubTypesComponentsSplit/RequestBodies.swift
   491 Sources/GithubTypesComponentsSplit/Responses.swift
311782 Sources/GithubTypesComponentsSplit/Schemas.swift
    15 Sources/GithubTypesComponentsSplit/Servers.swift
457230 total

% swift package clean && swift package update && swift build --skip-update --target GithubTypesComponentsSplit
Build complete! (422.18)

% wc -l Sources/GithubTypesSchemasSplit/*
 10492 Sources/GithubTypesSchemasSplit/APIProtocol.swift
    13 Sources/GithubTypesSchemasSplit/Components.swift
    30 Sources/GithubTypesSchemasSplit/Headers.swift
133735 Sources/GithubTypesSchemasSplit/Operations.swift
   657 Sources/GithubTypesSchemasSplit/Parameters.swift
    15 Sources/GithubTypesSchemasSplit/RequestBodies.swift
   491 Sources/GithubTypesSchemasSplit/Responses.swift
100277 Sources/GithubTypesSchemasSplit/Schemas.1.swift
100545 Sources/GithubTypesSchemasSplit/Schemas.2.swift
110984 Sources/GithubTypesSchemasSplit/Schemas.3.swift
    15 Sources/GithubTypesSchemasSplit/Schemas.swift
    15 Sources/GithubTypesSchemasSplit/Servers.swift
457269 total

% swift package clean && swift package update && swift build --skip-update --target GithubTypesSchemasSplit
Build complete! (140.95s)

% wc -l Sources/GithubTypes50kSplit/*
 10492 Sources/GithubTypes50kSplit/APIProtocol.swift
    13 Sources/GithubTypes50kSplit/Components.swift
    30 Sources/GithubTypes50kSplit/Headers.swift
 50029 Sources/GithubTypes50kSplit/Operations.1.swift
 50135 Sources/GithubTypes50kSplit/Operations.2.swift
 33596 Sources/GithubTypes50kSplit/Operations.3.swift
    13 Sources/GithubTypes50kSplit/Operations.swift
   657 Sources/GithubTypes50kSplit/Parameters.swift
    15 Sources/GithubTypes50kSplit/RequestBodies.swift
   491 Sources/GithubTypes50kSplit/Responses.swift
 50162 Sources/GithubTypes50kSplit/Schemas.1.1.swift
 50128 Sources/GithubTypes50kSplit/Schemas.1.2.swift
 50079 Sources/GithubTypes50kSplit/Schemas.2.1.swift
 50479 Sources/GithubTypes50kSplit/Schemas.2.2.swift
 55450 Sources/GithubTypes50kSplit/Schemas.3.1.swift
 55546 Sources/GithubTypes50kSplit/Schemas.3.2.swift
    15 Sources/GithubTypes50kSplit/Schemas.swift
    15 Sources/GithubTypes50kSplit/Servers.swift
457345 total

% swift package clean && swift package update && swift build --skip-update --target GithubTypes50kSplit
Build complete! (116.46s)

I think we have other features we can explore too (e.g. only generating the bits that users want to use), which will have much bigger impact, but I think we should keep this on the backlog.

[czechboy0]

Renaming as we should really try hard not to make this a config option.

[tonyarnold]

I'd love to see this — I setup the GitHub OpenAPI YAML spec in a standalone project, and the generated files took over 23 minutes to compile on an 16 core M3 Max with 64Gb RAM (2 of the performance cores were completely pegged, while the others did nothing).

I ended up needing to abandon the use of the Swift OpenAPI Generator packages for my project due to this issue.

Splitting the files up should result in dramatic compilation speedups under current Swift releases.

[simonjbeaumont]

@tonyarnold this is certainly something we can revisit post-1.0 as I believe our stance on the number of files, their names, and contents is not considered API.

In the meantime, have you tried adding a filter to the config file. You can read more about it here: https://swiftpackageindex.com/apple/swift-openapi-generator/0.3.5/documentation/swift-openapi-generator/soar-0008.

It even includes some examples using the GitHub API.