Skip to content

Use spray json like approach for config derivation with Pureconfig#4783

Closed
chetanmeh wants to merge 2 commits intoapache:masterfrom
chetanmeh:pureconfig-reflect
Closed

Use spray json like approach for config derivation with Pureconfig#4783
chetanmeh wants to merge 2 commits intoapache:masterfrom
chetanmeh:pureconfig-reflect

Conversation

@chetanmeh
Copy link
Member

This PR is based on work done as part of #4509 to figure out approaches to reduce compilation times. It uses a new proposed approach to switch to non Shapeless based config derivation similar to what we do with spary-json

Description

This PR is a work in progress till the new support in Pureconfig is completed. With new approach the compilation time for common/scala module reduced from ~ 60 sec to ~40 sec on my laptop

Related issue and scope

My changes affect the following components

  • API
  • Controller
  • Message Bus (e.g., Kafka)
  • Loadbalancer
  • Invoker
  • Intrinsic actions (e.g., sequences, conductors)
  • Data stores (e.g., CouchDB)
  • Tests
  • Deployment
  • CLI
  • General tooling
  • Documentation

Types of changes

  • Bug fix (generally a non-breaking change which closes an issue).
  • Enhancement or new feature (adds new functionality).
  • Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

  • I signed an Apache CLA.
  • I reviewed the style guides and followed the recommendations (Travis CI will check :).
  • I added tests to cover my changes.
  • My changes require further changes to the documentation.
  • I updated the documentation where necessary.

@chetanmeh
Copy link
Member Author

Below are the scalac profiling stats via -Ystatistics flag

Key aspects

  1. Time delta - Time reduced from 1 mins 12.067 secs to 47.163 secs
  2. Max time reduction was in typer phase from ~ 34s to ~15s

Current master

*** Cumulative timers for phases
#total compile time           : 1 spans, ()68896.897ms
  parser                      : 1 spans, ()858.851ms (1.2%)
  namer                       : 1 spans, ()273.763ms (0.4%)
  packageobjects              : 1 spans, ()3.263ms (0.0%)
  typer                       : 1 spans, ()34395.613ms (49.9%)
  patmat                      : 1 spans, ()4908.32ms (7.1%)
  superaccessors              : 1 spans, ()404.523ms (0.6%)
  extmethods                  : 1 spans, ()171.529ms (0.2%)
  pickler                     : 1 spans, ()225.027ms (0.3%)
  xsbt-api                    : 1 spans, ()3382.517ms (4.9%)
  xsbt-dependency             : 1 spans, ()1417.352ms (2.1%)
  refchecks                   : 1 spans, ()2281.169ms (3.3%)
  uncurry                     : 1 spans, ()2171.694ms (3.2%)
  fields                      : 1 spans, ()1780.857ms (2.6%)
  tailcalls                   : 1 spans, ()422.632ms (0.6%)
  specialize                  : 1 spans, ()1144.106ms (1.7%)
  explicitouter               : 1 spans, ()369.297ms (0.5%)
  erasure                     : 1 spans, ()4057.839ms (5.9%)
  posterasure                 : 1 spans, ()209.068ms (0.3%)
  lambdalift                  : 1 spans, ()432.697ms (0.6%)
  constructors                : 1 spans, ()318.683ms (0.5%)
  flatten                     : 1 spans, ()183.302ms (0.3%)
  mixin                       : 1 spans, ()1522.448ms (2.2%)
  cleanup                     : 1 spans, ()237.473ms (0.3%)
  delambdafy                  : 1 spans, ()621.277ms (0.9%)
  jvm                         : 1 spans, ()7024.781ms (10.2%)
  xsbt-analyzer               : 1 spans, ()22.947ms (0.0%)
Completed Scala compilation: 1 mins 12.067 secs

Post config switch

*** Cumulative timers for phases
#total compile time           : 1 spans, ()40351.618ms
  parser                      : 1 spans, ()2254.276ms (5.6%)
  namer                       : 1 spans, ()364.664ms (0.9%)
  packageobjects              : 1 spans, ()156.344ms (0.4%)
  typer                       : 1 spans, ()15632.499ms (38.7%)
  patmat                      : 1 spans, ()2686.365ms (6.7%)
  superaccessors              : 1 spans, ()292.228ms (0.7%)
  extmethods                  : 1 spans, ()115.676ms (0.3%)
  pickler                     : 1 spans, ()198.844ms (0.5%)
  xsbt-api                    : 1 spans, ()2639.931ms (6.5%)
  xsbt-dependency             : 1 spans, ()833.732ms (2.1%)
  refchecks                   : 1 spans, ()850.849ms (2.1%)
  uncurry                     : 1 spans, ()938.083ms (2.3%)
  fields                      : 1 spans, ()492.616ms (1.2%)
  tailcalls                   : 1 spans, ()257.213ms (0.6%)
  specialize                  : 1 spans, ()845.471ms (2.1%)
  explicitouter               : 1 spans, ()233.063ms (0.6%)
  erasure                     : 1 spans, ()2262.172ms (5.6%)
  posterasure                 : 1 spans, ()177.216ms (0.4%)
  lambdalift                  : 1 spans, ()397.438ms (1.0%)
  constructors                : 1 spans, ()247.595ms (0.6%)
  flatten                     : 1 spans, ()134.042ms (0.3%)
  mixin                       : 1 spans, ()1405.549ms (3.5%)
  cleanup                     : 1 spans, ()140.578ms (0.3%)
  delambdafy                  : 1 spans, ()403.887ms (1.0%)
  jvm                         : 1 spans, ()6313.895ms (15.6%)
  xsbt-analyzer               : 1 spans, ()20.235ms (0.1%)
Completed Scala compilation: 47.163 secs

Copy link
Member

@rabbah rabbah left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks worthwhile.

@chetanmeh
Copy link
Member Author

This needs to be put on hold for some time as no consensus was reached on Pureconfig side on used approach. Need to try a new simple macro based approach which looks like would take some time

@style95 style95 added the stale old issue which needs to validate label May 20, 2023
@style95
Copy link
Member

style95 commented May 20, 2023

@chetanmeh
I marked this PR as stale to close soon due to long inactivity.
But this is already approved and looks worthwhile.
Please let me know if you would work on it again.

@style95 style95 closed this May 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

stale old issue which needs to validate

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

Comments