Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(gen): slow examples generation #66

Closed
shadowspore opened this issue Nov 28, 2021 · 1 comment
Closed

perf(gen): slow examples generation #66

shadowspore opened this issue Nov 28, 2021 · 1 comment
Labels
area: gen Generator issues performance Performance issues

Comments

@shadowspore
Copy link
Member

shadowspore commented Nov 28, 2021

I was wondering why examples tooks around 9 seconds to generate (at least, on my machine).
Here's investigation results:

  1. Code formatting is slow.
  2. Template execution is sequential, but can be done in parallel.
  3. The go generate directive executes commands sequentially, but they also can be executed in parallel (affects to this case the most).

I did some profiling to measure how much time which generation step takes, and this is the results for github api spec:

format=true

Parsing:    0.07569065 seconds
Generation: 0.11895104 seconds
Writing:    2.01221638 seconds

format=false

Parsing:    0.07272715 seconds
Generation: 0.11632878 seconds
Writing:    0.50160086 seconds

Surprisingly, the code formatting is the slowest part of the whole project.
Writing source code without formatting is x3-x4 times faster.
For the smaller OpenAPI documents the difference is not that dramatic, but anyway... that's annoying.

The best solution is to generate already formatted code.
This completely removes the necessity of code formatting step, and allows to render templates into files directly without buffering.
But it will be tricky to do becase of templates which can call each other recursively N times (we should handle indentation somehow).
This can make these complex templates even more complex.

The next two issues, unlike the previous one, should be easy to fix.

@shadowspore shadowspore added the enhancement New feature or request label Nov 28, 2021
@ernado
Copy link
Member

ernado commented Nov 28, 2021

The best solution is to generate already formatted code.

This is actually more annoying to do than waiting for code generation.
We are operating on very big schemas in our example repos, 99.9% users are not affected.

It is pretty general problem for all code generators (and also linters), so I propose:

  1. Focus on solving this problem in general
  2. Optimize formatting
  3. Use concurrency
  4. And only then try to generate already formatted code

Anyway, this issue is low priority. IMO, problem with high resource consumption during compilation has bigger impact.

Also: fellows from ent proposed another idea: generate less code (e.g. use reflect-based json over fully code-generated, this can be done with jx too), and same points apply to such optimization.

@shadowspore shadowspore changed the title let's talk about performance! perf(gen): slow examples generation Dec 12, 2021
@tdakkota tdakkota added area: gen Generator issues performance Performance issues and removed enhancement New feature or request labels Jul 15, 2022
@ernado ernado closed this as completed Jun 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area: gen Generator issues performance Performance issues
Projects
None yet
Development

No branches or pull requests

3 participants