[shuffle] Correctly generate Typescript bindings to Move scripts #9398

dimroc · 2021-10-12T17:10:36Z

Motivation

Prior to this PR, typescript generation was broken with mismatched types, and incorrect imports.

Here, we correctly generate typescript bindings to move scripts, with zero manual tweaking and just-in-time (JIT) generate typescript libraries for interactions with move scripts, right as a developer enters the shuffle REPL.

Notable highlights:

Replaces javascript keywords like module with moduleName, and asserts the new ABI against a goldenfile.
Uses newer version of serde-generate with first class support for deno typescript runtime.
Adds a PeerToPeerScript typescript demo for TS runtime testing.
E2E tests typescript generation with script above.
Adds deno to the dev_setup.sh for codegen.
Fixes bug where BigInt was erroneously used as the primitive over bigint.
DenoFmt a bunch of temporarily checked in files. Will be deleted once the next JIT generate step is complete.
Refactor .ts helpers into its own file to prevent clobbering by codegen.
Typescript Codegen happens on shuffle new,console,test

Have you read the Contributing Guidelines on pull requests?

yes

Test Plan

cargo test -p transaction-builder-generator
cargo test -p shuffle

** Ignore files under generated/ from review.

Related PRs

Unblocks #9322, which adds a script function typescript demo.

language/transaction-builder/generator/tests/diem.goldenfile.yaml

dimroc · 2021-10-13T18:41:26Z

In order to make the PR smaller, I'll be adding the commit that JIT generates the typescript. That way, generated/ will be removed and the file count will be smaller.

libra-action · 2021-10-13T22:22:06Z

LEC dependency change summary:

target packages:
  M bcs 0.1.3 (direct third-party, crates.io)
    * version upgraded from 0.1.2
    * (unchanged features: [none])
  M serde-generate 0.20.2 (direct third-party, crates.io)
    * version upgraded from 0.19.1
    * (unchanged features: [none])
  M serde-reflection 0.3.4 (direct third-party, crates.io)
    * version upgraded from 0.3.2
    * (unchanged features: [none])

libra-action · 2021-10-13T22:22:07Z

LSR dependency change summary:

target packages:
  M bcs 0.1.3 (direct third-party, crates.io)
    * version upgraded from 0.1.2
    * (unchanged features: [none])

libra-action · 2021-10-13T22:22:09Z

release binaries dependency change summary:

target packages:
  M bcs 0.1.3 (direct third-party, crates.io)
    * version upgraded from 0.1.2
    * (unchanged features: [none])
  M serde-generate 0.20.2 (direct third-party, crates.io)
    * version upgraded from 0.19.1
    * (unchanged features: [none])
  M serde-reflection 0.3.4 (direct third-party, crates.io)
    * version upgraded from 0.3.2
    * (unchanged features: [none])

This is configured to run on CI via codegen-unit-test but ignored locally to keep test suite fast for other devs. Closes: #9398

github-actions · 2021-10-14T02:59:19Z

Forge Test Result

Forge test runner is terminated
Logs: https://diem-development.kb.us-west1.gcp.cloud.es.io:9243/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(),filters:!(),index:'2da2a6e0-f70a-11eb-ad53-b979a6c9b83a',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
Dashboard: http://mon.forge-0.aws.hlw3truzy4ls.com/d/overview/overview?from=1634179333188&to=1634180358474&orgId=1

❗ Forge Test failed - non-zero exit code for fgi

Repro cmd:

./scripts/fgi/run --tag land_987210db --base-image-tag land_b2f38ad1 --suite land_blocking_compat --report forge_report.json

github-actions · 2021-10-14T03:04:27Z

Cluster Test Result

Test runner setup time spent 281 secs
Compatibility test results for land_b2f38ad1 ==> land_987210db (PR)
1. All instances running land_b2f38ad1, generating some traffic on network
2. First full node land_b2f38ad1 ==> land_987210db, to validate new full node to old validator node traffic
3. First Validator node land_b2f38ad1 ==> land_987210db, to validate storage compatibility
4. First batch validators (14) land_b2f38ad1 ==> land_987210db, to test consensus and traffic between old full nodes and new validator node
5. First batch full nodes (14) land_b2f38ad1 ==> land_987210db
6. Second batch validators (15) land_b2f38ad1 ==> land_987210db, to upgrade rest of the validators
7. Second batch of full nodes (15) land_b2f38ad1 ==> land_987210db, to finish the network upgrade, time spent 643 secs
all up : 1133 TPS, 4008 ms latency, 4550 ms p99 latency, no expired txns, time spent 250 secs
Logs: http://kibana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T02:41:53Z',to:'2021-10-14T03:04:26Z'))
Dashboard: http://grafana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1634179313000&to=1634180666000
Validator 1 logs: http://kibana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T02:41:53Z',to:'2021-10-14T03:04:26Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_b2f38ad1 --cluster-test-tag land_987210db -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_987210db --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌

bors-libra · 2021-10-14T03:04:46Z

💔 Test Failed - ci-test

github-actions · 2021-10-14T03:57:39Z

Forge Test Result

Compatibility test results for land_b2f38ad1 ==> land_987210db (PR)
1. Downgrade all validators to older version: land_b2f38ad1
2. Upgrading first Validator to new version: land_987210db
3. Upgrading rest of first batch to new version: land_987210db
4. upgrading second batch to new version: land_987210db
5. check swarm health
Compatibility test for land_b2f38ad1 ==> land_987210db passed
performance::performance-benchmark : 622 TPS, 7262 ms latency, 9550 ms p99 latency,no expired txns
Logs: https://diem-development.kb.us-west1.gcp.cloud.es.io:9243/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(),filters:!(),index:'2da2a6e0-f70a-11eb-ad53-b979a6c9b83a',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
Dashboard: http://mon.forge-1.aws.hlw3truzy4ls.com/d/overview/overview?from=1634182705779&to=1634183858376&orgId=1

Repro cmd:

./scripts/fgi/run --tag land_987210db --base-image-tag land_b2f38ad1 --suite land_blocking_compat --report forge_report.json

🎉 Land-blocking forge test passed! 👌

github-actions · 2021-10-14T04:03:41Z

Cluster Test Result

Test runner setup time spent 289 secs
Compatibility test results for land_b2f38ad1 ==> land_987210db (PR)
1. All instances running land_b2f38ad1, generating some traffic on network
2. First full node land_b2f38ad1 ==> land_987210db, to validate new full node to old validator node traffic
3. First Validator node land_b2f38ad1 ==> land_987210db, to validate storage compatibility
4. First batch validators (14) land_b2f38ad1 ==> land_987210db, to test consensus and traffic between old full nodes and new validator node
5. First batch full nodes (14) land_b2f38ad1 ==> land_987210db
6. Second batch validators (15) land_b2f38ad1 ==> land_987210db, to upgrade rest of the validators
7. Second batch of full nodes (15) land_b2f38ad1 ==> land_987210db, to finish the network upgrade, time spent 695 secs
all up : 1125 TPS, 4030 ms latency, 4600 ms p99 latency, no expired txns, time spent 250 secs
Logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T03:37:59Z',to:'2021-10-14T04:03:40Z'))
Dashboard: http://grafana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1634182679000&to=1634184220000
Validator 1 logs: http://kibana.ct-0-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T03:37:59Z',to:'2021-10-14T04:03:40Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_b2f38ad1 --cluster-test-tag land_987210db -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_987210db --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌

dimroc · 2021-10-14T04:10:26Z

/land

./scripts/dev_setup.sh -s

This is configured to run on CI via codegen-unit-test but ignored locally to keep test suite fast for other devs. Closes: diem#9398

github-actions · 2021-10-14T07:20:32Z

Forge Test Result

Compatibility test results for land_b580d224 ==> land_c800ee12 (PR)
1. Downgrade all validators to older version: land_b580d224
2. Upgrading first Validator to new version: land_c800ee12
3. Upgrading rest of first batch to new version: land_c800ee12
4. upgrading second batch to new version: land_c800ee12
5. check swarm health
Compatibility test for land_b580d224 ==> land_c800ee12 passed
performance::performance-benchmark : 626 TPS, 7226 ms latency, 9150 ms p99 latency,no expired txns
Logs: https://diem-development.kb.us-west1.gcp.cloud.es.io:9243/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(),filters:!(),index:'2da2a6e0-f70a-11eb-ad53-b979a6c9b83a',interval:auto,query:(language:kuery,query:''),sort:!(!('@timestamp',desc)))
Dashboard: http://mon.forge-0.aws.hlw3truzy4ls.com/d/overview/overview?from=1634194878980&to=1634196031422&orgId=1

Repro cmd:

./scripts/fgi/run --tag land_c800ee12 --base-image-tag land_b580d224 --suite land_blocking_compat --report forge_report.json

🎉 Land-blocking forge test passed! 👌

github-actions · 2021-10-14T07:23:10Z

Cluster Test Result

Test runner setup time spent 258 secs
Compatibility test results for land_b580d224 ==> land_c800ee12 (PR)
1. All instances running land_b580d224, generating some traffic on network
2. First full node land_b580d224 ==> land_c800ee12, to validate new full node to old validator node traffic
3. First Validator node land_b580d224 ==> land_c800ee12, to validate storage compatibility
4. First batch validators (14) land_b580d224 ==> land_c800ee12, to test consensus and traffic between old full nodes and new validator node
5. First batch full nodes (14) land_b580d224 ==> land_c800ee12
6. Second batch validators (15) land_b580d224 ==> land_c800ee12, to upgrade rest of the validators
7. Second batch of full nodes (15) land_b580d224 ==> land_c800ee12, to finish the network upgrade, time spent 650 secs
all up : 1148 TPS, 3959 ms latency, 4500 ms p99 latency, no expired txns, time spent 250 secs
Logs: http://kibana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T07:00:56Z',to:'2021-10-14T07:23:09Z'))
Dashboard: http://grafana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/d/performance/performance?from=1634194856000&to=1634196189000
Validator 1 logs: http://kibana.ct-2-k8s-testnet.aws.hlw3truzy4ls.com/app/kibana#/discover?_g=(time:(from:'2021-10-14T07:00:56Z',to:'2021-10-14T07:23:09Z'))&_a=(columns:!(log),query:(language:kuery,query:'kubernetes.pod_name:"val-1"'),sort:!(!('@timestamp',desc)))

Repro cmd:

./scripts/cti --tag land_b580d224 --cluster-test-tag land_c800ee12 -E BATCH_SIZE=15 -E UPDATE_TO_TAG=land_c800ee12 --report report.json --suite land_blocking_compat

🎉 Land-blocking cluster test passed! 👌

libra-action · 2021-10-14T07:26:55Z

Dependency update review

bcs updated: 0.1.2 --> 0.1.3

No known advisories	✅
No change in the build script	✅
No change in any file with unsafe code	✅
The source and crates.io code are the same	✅

Cilck to show details

Click to show version diff summary

total files changed	6
total rust files changed	3
total loc change	36

Click to show changed files

tests/serde.rs
src/ser.rs
Cargo.toml
Cargo.toml.orig
.cargo_vcs_info.json
src/de.rs

serde-generate updated: 0.19.1 --> 0.20.2

No known advisories	✅
No change in the build script	✅
No change in any file with unsafe code	✅
The source and crates.io code are the same	✅

Cilck to show details

Click to show version diff summary

total files changed	39
total rust files changed	21
total loc change	1,392

Click to show changed files

runtime/typescript/serde/types.ts
tests/python_runtime.rs
runtime/typescript/bincode/mod.ts
runtime/typescript/serde/serializer.ts
tests/cpp_runtime.rs
README.md
runtime/typescript/serde/deserializer.ts
tests/rust_generation.rs
runtime/typescript/bcs/bcsDeserializer.ts
runtime/typescript/serde/mod.ts
src/lib.rs
src/python3.rs
runtime/typescript/bincode/bincodeSerializer.ts
tests/cpp_generation.rs
src/cpp.rs
tests/java_generation.rs
src/test_utils.rs
tests/rust_runtime.rs
runtime/typescript/serde/binaryDeserializer.ts
tests/typescript_runtime.rs
tests/golang_runtime.rs
runtime/typescript/serde/binarySerializer.ts
runtime/typescript/bcs/mod.ts
Cargo.lock
tests/csharp_generation.rs
.cargo_vcs_info.json
Cargo.toml
tests/golang_generation.rs
src/rust.rs
src/typescript.rs
tests/java_runtime.rs
tests/typescript_generation.rs
runtime/typescript/bcs/bcsSerializer.ts
Cargo.toml.orig
runtime/typescript/bincode/bincodeDeserializer.ts
tests/python_generation.rs
src/csharp.rs
tests/csharp_runtime.rs
runtime/typescript/bincode/index.ts

serde-reflection updated: 0.3.2 --> 0.3.4

No known advisories	✅
No change in the build script	✅
No change in any file with unsafe code	✅
The source and crates.io code are the same	❌

Cilck to show details

Click to show version diff summary

total files changed	10
total rust files changed	6
total loc change	486

Click to show changed files

README.md
src/ser.rs
tests/format.rs
.cargo_vcs_info.json
Cargo.toml.orig
Cargo.toml
src/format.rs
src/trace.rs
src/lib.rs
tests/serde.rs

Click to show the files that differ in crates.io from the git source

src/lib.rs

bors-libra added this to In Review in bors Oct 12, 2021

diem-cla-bot bot added the cla-signed label Oct 12, 2021

dimroc force-pushed the typescript-deno-txn-builders-scripts branch from 9c1c930 to 99315e2 Compare October 12, 2021 17:37

libra-action added tcb-deps-changed deps-changed labels Oct 12, 2021

dimroc force-pushed the typescript-deno-txn-builders-scripts branch 2 times, most recently from d012401 to ea0efb8 Compare October 12, 2021 19:57

dimroc changed the title ~~JIT generate Typescript bindings to Move scripts~~ [shuffle] JIT generate Typescript bindings to Move scripts Oct 12, 2021

dimroc changed the title ~~[shuffle] JIT generate Typescript bindings to Move scripts~~ [shuffle] Correctly generate Typescript bindings to Move scripts Oct 12, 2021

dimroc force-pushed the typescript-deno-txn-builders-scripts branch 4 times, most recently from deb03dc to dce7a0e Compare October 13, 2021 01:16

dimroc requested review from ma2bd and sunmilee October 13, 2021 01:16

dimroc marked this pull request as ready for review October 13, 2021 02:50

dimroc requested a review from a team October 13, 2021 02:50

dimroc commented Oct 13, 2021

View reviewed changes

language/transaction-builder/generator/tests/diem.goldenfile.yaml Outdated Show resolved Hide resolved

dimroc marked this pull request as draft October 13, 2021 19:25

dimroc force-pushed the typescript-deno-txn-builders-scripts branch 4 times, most recently from a17526e to 0d45797 Compare October 13, 2021 22:01

dimroc marked this pull request as ready for review October 13, 2021 22:04

dimroc force-pushed the typescript-deno-txn-builders-scripts branch from 0d45797 to dc69f1d Compare October 13, 2021 22:16

xli approved these changes Oct 13, 2021

View reviewed changes

bors-libra pushed a commit that referenced this pull request Oct 14, 2021

[shuffle] #[ignore] TS generation test w deno runtime:

987210d

This is configured to run on CI via codegen-unit-test but ignored locally to keep test suite fast for other devs. Closes: #9398

bors-libra moved this from Queued to Testing in bors Oct 14, 2021

bors-libra moved this from Testing to In Review in bors Oct 14, 2021

bors-libra moved this from In Review to Queued in bors Oct 14, 2021

dimroc added 7 commits October 14, 2021 06:33

[shuffle] remove keywords from TS txn builder generation

f631889

[shuffle] isolate generated txn builders into generated/

97f25b7

[shuffle] add deno to scripts/dev_setup.sh for codegen:

bdfbe6d

./scripts/dev_setup.sh -s

[shuffle] add P2P demo integration test

0c1be8b

[shuffle] remove address hardcodings from typescript helpers

f169031

[shuffle] JIT generate typescript libraries for Move pkg

9535d1b

[shuffle] #[ignore] TS generation test w deno runtime:

c800ee1

This is configured to run on CI via codegen-unit-test but ignored locally to keep test suite fast for other devs. Closes: diem#9398

bors-libra moved this from Queued to Testing in bors Oct 14, 2021

bors-libra dismissed xli’s stale review via c800ee1 October 14, 2021 07:23

bors-libra force-pushed the typescript-deno-txn-builders-scripts branch from dc69f1d to c800ee1 Compare October 14, 2021 07:23

bors-libra removed this from Testing in bors Oct 14, 2021

bors-libra closed this in c800ee1 Oct 14, 2021

bors-libra merged commit c800ee1 into diem:main Oct 14, 2021

bors-libra temporarily deployed to Sccache October 14, 2021 07:23 Inactive

bors-libra temporarily deployed to Docker October 14, 2021 07:24 Inactive

bors-libra temporarily deployed to Sccache October 14, 2021 07:24 Inactive

bors-libra temporarily deployed to Docker October 14, 2021 07:24 Inactive

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[shuffle] Correctly generate Typescript bindings to Move scripts #9398

[shuffle] Correctly generate Typescript bindings to Move scripts #9398

dimroc commented Oct 12, 2021 •

edited

dimroc commented Oct 13, 2021

libra-action commented Oct 13, 2021

libra-action commented Oct 13, 2021

libra-action commented Oct 13, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

bors-libra commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

dimroc commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

libra-action commented Oct 14, 2021

[shuffle] Correctly generate Typescript bindings to Move scripts #9398

[shuffle] Correctly generate Typescript bindings to Move scripts #9398

Conversation

dimroc commented Oct 12, 2021 • edited

Motivation

Have you read the Contributing Guidelines on pull requests?

Test Plan

Related PRs

dimroc commented Oct 13, 2021

libra-action commented Oct 13, 2021

libra-action commented Oct 13, 2021

libra-action commented Oct 13, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

bors-libra commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

dimroc commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

github-actions bot commented Oct 14, 2021

libra-action commented Oct 14, 2021

Dependency update review

bcs updated: 0.1.2 --> 0.1.3

serde-generate updated: 0.19.1 --> 0.20.2

serde-reflection updated: 0.3.2 --> 0.3.4

dimroc commented Oct 12, 2021 •

edited