Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use JS IR backend and generate TypeScript sources #427

Merged
merged 25 commits into from
Apr 2, 2023

Conversation

Whathecode
Copy link
Member

@Whathecode Whathecode commented Jan 9, 2023

This is a proof-of-concept to show how generated TypeScript (TS) sources for this project can be augmented with manually written TS declarations to:

  • export types from external libraries which aren't exported (no @JsExport applied)
  • export types which can't be exported yet (@JsExport can't be applied due to current Kotlin JS limitations)

This PR replaces a previous attempt at doing the same on Kotlin 1.6.10, with an approach which works for Kotlin 1.8.0.

This is mostly a hack, but shows that this is feasible. In short:

  • The publish-npm-packages project depends on the projects which need to be exported and includes JsExport on classes which ensure that APIs which need to be made publicly available aren't optimized out. The JS output of this project can also be used to look up mangled method names.
  • copyTestJsSources in build.gradle copies the build output of publish-npm-packages into node modules (no package.json for publication created yet)
  • Copied files are modified to:
    • Refer to node modules instead of relative file paths.
    • Types which are used across modules but aren't JS exported are exported through $_$ using mangled names. Include additional exports of named internal types needed for TS augmentations. These are specified in the publish-npm-packages/src/forced-exports folder in a separate file per module.
  • typescript-declarations/@types contains TS declarations for the named types forced to be exported in forced-exports.
  • typescript-declarations/src contains facades exporting the types used for internal communication by augmenting them with clean interfaces in the expected namespaces and adding facade methods with clean names which map to the mangled ones.

The unit tests which were implemented for the JS Legacy backend now pass on the JS IR backend with the augmented non-mangled TS declarations. Unmangled facades with sufficient functionality to use the CARP APIs have been added for the Kotlin standard library, kotlinx datetime, and kotlinx serialization. Regex could be exported similarly as well, but there is no real use since this is not exposed in relevant APIs for web clients and only used for internal logic.

With these APIs made available in the expected namespaces, the next step is to:

  • apply another post-compilation step which replaces non-exportable types, which are represented as any followed by their Kotlin type in a comment in TypeScript, with the actual type for all augmented types made available through declaration merging.

@JsExport, where possible, has been applied to CARP:

  • common
  • studies
  • protocols
  • deployment
  • data

Serializers haven't been exported since they are already available through the companion object. The following types @JsExport couldn't be applied to due to compiler warnings.

CARP common:

  • ApplicationService (nice to have, but not needed)
  • SamplingConfigurationBuilder (no real need)
  • SamplingConfigurationMapBuilder (no real need)
  • ApplicationServiceEventBus (full file, no real need)
  • EventBus (full file, no real need)
  • RangeExtensions (extension functions can be exported, but no need and increases JS size)

For all services in deployments/studies/protocols/data subsystems, due to Kotlin JS compiler bug:

  • XYZService (nice to have, but not needed)
  • XYZServiceHost (not needed)

Augmentations to work around non-exportable types have been applied to types in CARP:

  • common
  • studies
  • protocols
  • deployment
  • data

All TypeScript declarations and unit tests were removed since they need to be rewritten to match the new JS backend. This is already done for the Kotlin stdlib. But, to make this work on the IR backend, a facade for the Kotlin standard library is included, along with a build setup to hack into the generated Javascript sources to make this work.
When methods were overloaded, one module may use a different internal name than another, causing the current renaming of import/export to fail.
Instead, keep the original compiled import/export names, but add the custom desired ones.
This doesn't export all types in common yet. Only those used in the TS unit tests which were present before.

Because additional code was exported, the mangled names in the kotlin standard library and datetime library had to be updated.
Again, because additional code was exported, the mangled names in the kotlin standard library and datetime library had to be updated.
Before, while the code worked, trying to do typecasts etc. would fail since the types weren't exported. By exporting the namespaces within the facade as opposed to in the declaration, this seems to be solved.
…n generated TS

Types for which no `@JsExport` is applied, when exposed in an API, are exported as `any`. For those types for which manual facades are written, replace these in the generated TypeScript sources with their facade type.
…ubmodules

This does not re-introduce TypeScript unit tests yet.
This doesn't include request object tests which were there before because so far these fail to be exported.
@Whathecode Whathecode force-pushed the js-ir-backend-1.8 branch 2 times, most recently from 83f6199 to 4276bb8 Compare March 12, 2023 20:07
This is no longer used and is superseded by `ApplicationServiceInvoker`.
@Whathecode
Copy link
Member Author

Just managed to get a big blocker out of the way. Application service request objects are now exported to JS!

This didn't work before due to a compilation error: https://youtrack.jetbrains.com/issue/KT-57356

But, by removing `@JsExport` from `ApplicationService`, where it isn't really needed, compilation succeeds. As part of this, old TypeScript unit tests operating on request objects were reintroduced.

This again changed the JS export drastically enough so that the custom typescript declarations hacking into non-exported JS exports had to be updated.
`WebTask` export didn't work on JS due to a bug in generated TypeScript sources: required parameters can't follow optional parameters: https://youtrack.jetbrains.com/issue/KT-53180/Kotlin-JS-generated-TypeScript-constructor-can-have-TS1016-A-required-parameter-cannot-follow-an-optional-parameter-error-with
`AccountIdentity` couldn't be exported due to the companion object on the interface: https://youtrack.jetbrains.com/issue/KT-51292/Proposed-behavior-of-JsExport-on-interfaces-and-classes-with-companion-objects
Using an abstract class is a good enough solution for now.
JS export of `Smartphone` didn't work due to a bug: required parameters can't follow optional parameters: https://youtrack.jetbrains.com/issue/KT-53180/

But, using the constructor for the builder pattern wasn't ideal either way. With the previous builder in the constructor, to access `isOptional`, or later subbuilders once we add them (like `buildSamplingConfiguration()`), the builder would need to be applied multiple times.
Unclear why this failed before. Maybe a clean was needed, or something got fixed in Kotlin 1.8.10 which was broken on 1.8.0 when I last tried this.
Copy link
Collaborator

@xelahalo xelahalo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The generated sources look sweet 🤙

Now the only thing that would be nice is a way for us to access the sources through NPM. Though I think I could easily do that for you, I've done something similar with GitHub packages.

@Whathecode Whathecode merged commit 7feb795 into develop Apr 2, 2023
@Whathecode Whathecode deleted the js-ir-backend-1.8 branch April 2, 2023 10:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants