Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[spec] Tweak tokenisation for text format #1499

Merged
merged 2 commits into from
Jun 28, 2022
Merged

[spec] Tweak tokenisation for text format #1499

merged 2 commits into from
Jun 28, 2022

Conversation

rossberg
Copy link
Member

@rossberg rossberg commented Jun 28, 2022 •
edited

As discussed at the last meeting, this disallows omission of spacing around string tokens and adds a tokenisation test. Supersedes #1490.

@rossberg rossberg requested a review from tlively June 28, 2022 13:09
rossberg
rossberg commented Jun 28, 2022
View reviewed changes
test/core/tokens.wast
)

(module
(data $l "")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, apparently, GH can't display the Unicode characters used here and below, at least not on my browser.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see two little box things, fwiw. What character is it supposed to be?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I don't know. I just copied it from comments.wast, but my editor only shows boxes as well. :)

@rossberg rossberg mentioned this pull request Jun 28, 2022
Closed
tlively
tlively reviewed Jun 28, 2022
View reviewed changes
Copy link
Member

@tlively tlively left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it a problem that this wouldn't handle cases with unpaired double quotes?

test/core/tokens.wast
)

(module
(data $l "")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see two little box things, fwiw. What character is it supposed to be?

rossberg
rossberg commented Jun 28, 2022
View reviewed changes
Copy link
Member Author

@rossberg rossberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, that's fine, that's illegal and will create a lexing error as before. For the purpose at hand, we only have to make sure that any relevant sequence of legal tokens is considered a single token.

test/core/tokens.wast
)

(module
(data $l "")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be honest, I don't know. I just copied it from comments.wast, but my editor only shows boxes as well. :)

tlively
tlively approved these changes Jun 28, 2022
View reviewed changes
Copy link
Member

@tlively tlively left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@rossberg rossberg merged commit e0a6a31 into main Jun 28, 2022
@rossberg rossberg deleted the tokenisation branch June 28, 2022 20:50
@keithw keithw mentioned this pull request Sep 21, 2022
Merged
keithw added a commit to WebAssembly/wabt that referenced this pull request Oct 3, 2022
@keithw
Add tokens test + adjust Wast lexing to match updated spec (#2001)
a2c0d17 
* Update testsuite (adding new tokens.txt test)

* Adjust Wast lexing to match updated spec (WebAssembly/spec#1499)
@keithw keithw mentioned this pull request Oct 3, 2022
Merged
matthias-blume pushed a commit to matthias-blume/wabt that referenced this pull request Dec 16, 2022
@keithw @matthias-blume
Add tokens test + adjust Wast lexing to match updated spec (WebAssemb…
71d5ec5 
…ly#2001)

* Update testsuite (adding new tokens.txt test)

* Adjust Wast lexing to match updated spec (WebAssembly/spec#1499)
keithw added a commit to WebAssembly/wabt that referenced this pull request Feb 25, 2023
@keithw
Improve parsing of invalid annotations
478b0a9 
This had been crashing even with annotations disabled.
Adds a regression test.

This was missed in #2001 when updating the parser to match
the updated spec (WebAssembly/spec#1499).

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53935
@keithw keithw mentioned this pull request Feb 25, 2023
Merged
keithw added a commit to WebAssembly/wabt that referenced this pull request Feb 25, 2023
@keithw
Improve parsing of invalid annotations
828277f 
This had been crashing even with annotations disabled.
Adds a regression test.

This was missed in #2001 when updating the parser to match
the updated spec (WebAssembly/spec#1499).

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53935
keithw added a commit to WebAssembly/wabt that referenced this pull request Feb 25, 2023
@keithw
Improve lexing of invalid annotations (#2150)
3d9ec15 
This had been crashing even with annotations disabled.
Adds a regression test.

This was missed in #2001 when updating the lexer to match
the updated spec (WebAssembly/spec#1499).

Fixes https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=53935
dhil pushed a commit to dhil/webassembly-spec that referenced this pull request Mar 2, 2023
@rossberg
[spec] Tweak tokenisation for text format (WebAssembly#1499)
11d1a53
dhil added a commit to effect-handlers/wasm-spec that referenced this pull request Mar 17, 2023
@dhil @rossberg
@hanshoglund @ngzhian @sbc100 @tlively @ggreif @codefromthecrypt @titzer @keithw @gahaas @wooster0 @tpmccallum @ia0 @vapourismo @JamesMenetrey @ivan-pan @ictrobot @aathan @fifofefe @mnordine @CosineP @ariez-xyz @surma @takikawa @ianh @tomstuart @Jamesernator @whirlicote @Ms2ger @Adzz @osa1 @eutro @michaelficarra @yamt @candymate @bonjune @ShinWonho @suhdonghwi @jimblandy @aheejin @littledan
Merge pull request #25 from dhil/wasmfx-funcref-merge
9f861ea 
* [spec] Add reference types to overview (WebAssembly#1394)

* [interpreter] Remove use of physical equality on characters (WebAssembly#1396)

* Merge SIMD proposal (WebAssembly#1391)

SIMD is [phase 5](WebAssembly/simd#507), merge all the changes back into main spec.

* Remove merge conflict marker

* [spec] Handle v128 in validation algorithm (WebAssembly#1399)

* [spec] Fix instruction table (WebAssembly#1402)

* Add tests for functions without end marker. NFC (WebAssembly#1405)

Inspired by this downstream test in wabt:
WebAssembly/wabt#1775

Fixes: WebAssembly#1404

* Describe correct tail call behavior across modules

Whether tail calls across module boundaries would guarantee tail call behavior was previously an open question, but @thibaudmichaud confirmed that they would guarantee tail call behavior in V8 in WebAssembly/tail-call#15 (comment).

* [interpreter] Fix a typo in README (WebAssembly#1406)

* Add a link to the proposals repo (WebAssembly#1409)

Fixes WebAssembly#1407.

* [spec] Add note regarding parameter names (WebAssembly#1412)

* Comments WIP

* Merge upstream (WebAssembly#55)

* Typo

* [spec] Clarifying note on text format (WebAssembly#1420)

Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Andreas Rossberg <rossberg@mpi-sws.org>

* Eps

* Eps

* [test] Fix section size in binary test (WebAssembly#1424)

* Update document README to install six

* Disallow type recursion (WebAssembly#56)

* Fix import order

* Add --generate-js-only flag to test runner

This will return early right after generating JS from the wast test
files. It will not attempt to run the tests, or do the round trip
conversion from wasm <-> wast.

This is convenient for proposals to add tests without having to
update the reference interpreter with implementation, and generate those
tests to JS to run in other Wasm engines.

Fixes WebAssembly#1430.

* Remove use of let from func.bind test

* Add call3

* [spec] Fix missing mention of vectype (WebAssembly#1436)

Fixed WebAssembly#1435.

* [spec] Fix single-table limitation in module instantiation (WebAssembly#1434)

* [spec] Fix missing immediate on table.set (WebAssembly#1441)

* [docs] Update syntax in examples (WebAssembly#1442)

* Clarification in proposals README

* [interpreter] Tweak start section AST to match spec

* [spec] Bump release to 2 (WebAssembly#1443)

At yesterday's WG meeting, we decided to make a new release, now switching to the Evergreen model. For administrative and technical reasons having to do with W3C procedure, we decided to bump the release number to 2.

From now on, the standard will iterate at version 2 from the W3C's official perspective. We use minor release numbers internally to distinguish different iterations.

(@ericprud, I hope I understood correctly that the Bikeshed "level" also needed to be bumped to 2.)

* Remove test cases with let

* Sync wpt test (WebAssembly#1449)

* [spec] Fix typo (WebAssembly#1448)

* [proposals] Add missing start to example (WebAssembly#1454)

* [spec] "version 2.0" -> "release 2.0" (WebAssembly#1452)

* [spec] Fix typo (WebAssembly#1458)

* [test] Add assert_trap for unreached valid case (WebAssembly#1460)

* [interpreter] Name the type Utf8.unicode

* [spec] Fix binary format of data/elem tags to allow LEB (WebAssembly#1461)

* [spec] Fix typos in numeric operations (WebAssembly#1467)

* [spec] Fix syntax error in element segments validation rule (WebAssembly#1465)

* [spec] Fix typo in global instance syntax (WebAssembly#1466)

* [spec] Fix typos in module instantiation (WebAssembly#1468)

* [interpreter] Turn into a Dune package (WebAssembly#1459)

* [spec] Fix typos in instruction validation rules (WebAssembly#1462)

* [bib] Update latex .bib file for webassembly 2.0 (WebAssembly#1463)

* [spec] Add missing default for vector types (WebAssembly#1464)

* [spec] Fix typos in binary and text formats (WebAssembly#1469)

* [spec] Fix various typos (WebAssembly#1470)

* TypeError for Global constructor with v128

At the moment the spec requires a `LinkError` to be thrown when the `WebAssembly.Global` constructor is called for type `v128`. This was introduced in WebAssembly/simd#360, but according to the PR description, actually a `TypeError` should be thrown. The PR refers to https://github.com/WebAssembly/simd/blob/master/proposals/simd/SIMD.md#javascript-api-and-simd-values, and there a `TypeError` is required.

* [spec] Fix LEB opcodes in instruction index (WebAssembly#1475)

* [spec] Fix v128.loadX_splat in instruction index (WebAssembly#1477)

* [interpreter] Dune test suite (WebAssembly#1478)

* [interpreter] Fix warning flags for OCaml 4.13 (WebAssembly#1481)

* [interpreter] Simplify lexer and avoid table overflow on some architectures (WebAssembly#1482)

* [spec] Editorial nit (WebAssembly#1484)

* [interpreter] Produce error messages in encoder (WebAssembly#1488)

* [spec] Add missing close paren on table abbreviation (WebAssembly#1486)

Also remove an unnecessary space in the previous table abbreviation.

* [spec] Remove outdated note (WebAssembly#1491)

* Eps

* [interpreter] Factor data and element segments into abstract types (WebAssembly#1492)

* [spec] Update note on module initialization trapping (WebAssembly#1493)

* Fix type equality

* Fix typo

* [spec] Add note about control stack invariant to algorithm (WebAssembly#1498)

* [spec] Tweak tokenisation for text format (WebAssembly#1499)

* [test] Use still-illegal opcode (func-refs) (WebAssembly#1501)

* Fix minor typos and consistency issues in the validation algorithm. (WebAssembly#61)

* Add definition of defaultable types (WebAssembly#62)

No rules for locals yet, since those are still being discussed.

* Remove func.bind (WebAssembly#64)

* Implement 1a (WebAssembly#63)

* Subtyping on vector types & heap bottom check (WebAssembly#66)

* [interpreter] Bring AST closer to spec

* [spec] Fix typo (WebAssembly#1508)

* WIP

* Remove polymorphic variants

* Minor syntactic consistency fix (WebAssembly#68)

Change pseudo equality operator from `==` to `=`.

* Bump version

* [spec] Fix table.copy validation typo (WebAssembly#1511)

* More fixes

* Fix Latex

* Adjust intro

* Spec local initialization (WebAssembly#67)

* Add table initialiser (WebAssembly#65)

* [spec] Remove outdated note (WebAssembly#1491)

* [interpreter] Factor data and element segments into abstract types (WebAssembly#1492)

* [spec] Update note on module initialization trapping (WebAssembly#1493)

* [spec] Add note about control stack invariant to algorithm (WebAssembly#1498)

* [spec] Tweak tokenisation for text format (WebAssembly#1499)

* [test] Use still-illegal opcode (func-refs) (WebAssembly#1501)

* [spec] Fix typo (WebAssembly#1508)

* [spec] Fix table.copy validation typo (WebAssembly#1511)

* Merge fallout

* Latex fixes

* [spec] Minor copy edit (WebAssembly#1512)

* Spec changelog

* [spec] Trivial editorial fix

* Update embedding

* Oops

* Argh

* Rename Sem to Dyn

* Readd match.mli

* [interpreter] Build wast.js with Js_of_ocaml (WebAssembly#1507)

* [interpreter] Add flag for controlling call budget

* Spec zero byte

* Fix table/elem expansion (WebAssembly#71)

* Fix merge artefact

* Restrict init from stack-polymorphism (WebAssembly#75)

* [spec] Simplify exec rule for if (WebAssembly#1517)

* [spec] Formatting tweak (WebAssembly#1519)

* [spec] Fix typing rule in appendix (WebAssembly#1516)

* [spec] Fix “invertible” typo (WebAssembly#1520)

* [spec] Correct use of opdtype and stacktype (WebAssembly#1524)

* [spec] Add note to instruction index (WebAssembly#1528)

* Add type annotation to call_ref (WebAssembly#76)

* [spec] Tweak wording to avoid first person

* Eps

* Eps2

* Eps3

* Remove unneeded assumption type

* [spec/test] Fix scoping of non-imported globals (WebAssembly#1525)

* Fix test

* A couple of tests

* Performance improvement

* Typo

* Another typo

* [spec] Fix language config

* Fix null subtyping being wrong way around (WebAssembly#79)

* [spec] Fix naming typo (WebAssembly#1532)

* Defunctorise types again

* [spec] Add citation for WasmCert (WebAssembly#1533)

* [test] Fix async_index.js

* [test] Enable the i64 tests in imports.wast.

Fixes WebAssembly#1514.

* Minor tweak

* [js-api][web-api] Editorial: Fix some minor issues.

Fixes WebAssembly#1064.

* Update README.md (WebAssembly#1540)

Improve wording.

* [spec] Fix typo in element execution (WebAssembly#1544)

* [spec] Remove obsolete note (WebAssembly#1545)

* cccccc[klghketetivvtnnhvntikigrnueuhdkkukljgjuest/meta/generate_*.js: sync upstream JS with tests (WebAssembly#1546)

* [spec] Editorial tweak

* [test] test segment/table mismatch and externref segment (WebAssembly#1547)

* [interpreter] Remove duplicate token declarations (WebAssembly#1548)

* Update Soundness appendix (WebAssembly#72)

* [spec] Formatting eps

* Remove oboslete note in README (WebAssembly#82)

* Add `print_i64` to generated spec tests

WebAssembly@82a613d added `print_i64` to the standalone test files, but not to the ones generated by the spec interpreter.

* [test] Tweak binary-leb128 and simd_lane (WebAssembly#1555)

* [spec] Allow explicit keyword definitions (WebAssembly#1553)

Rather than describing keyword tokens as always being defined implicitly by terminal symbols in syntactic productions, describe them as being defined implicitly or explicitly. This accounts for the explicit definitions of `offset` and `align` phrases, which are lexically keywords, later in the chapter.

Fixes WebAssembly#1552.

* [js-api] editorial: adjust link for v128 type

* Factor local init tests to local_init.wast; add more (WebAssembly#84)

* Update JS API for no-frills

* [spec] Add missing case for declarative elem segments

Fixes WebAssembly#1562.

* [spec] Hotfix last accidental commit

* [spec] Fix hyperref (WebAssembly#1563)

* [spec] Bump sphinx version to fix Python problem

* [spec] Fix minor errors and inconsistencies (WebAssembly#1564)

* Spacing

* Fix a couple more superfluous brackets

* [spec] Eps

* [interpreter] Refactor parser to handle select & call_indirect correctly (WebAssembly#1567)

* [spec] Remove dead piece of grammar

* [test] elem.wast: force to use exprs in a element (WebAssembly#1561)

* Fix typos in SIMD exec/instructions

* Update interpreter README (WebAssembly#1571)

It previously stated that the formal spec did not exist, but the spec has existed for years now.

* [spec] Remove an obsolete exec step (WebAssembly#1580)

* [test] Optional tableidx for table.{get,set,size,grow,fill} (WebAssembly#1582)

* [spec] Fix abstract grammar for const immediate (WebAssembly#1577)

* [spec] Fix context composition in text format (WebAssembly#1578)

* [spec] Fix label shadowing (WebAssembly#1579)

* Try bumping OCaml

* Try bumping checkout

* Adjust for multi-return

* Tweak reduction rules

* Spec return_call_ref

* Fix

* Text format

* [spec] Fix typos in instruction index (WebAssembly#1584)

* [spec] Fix typo (WebAssembly#1587)

* [spec] Remove inconsistent newline (WebAssembly#1589)

* [interpreter] Remove legacy bigarray linking (WebAssembly#1593)

* [spec] Show scrolls for overflow math blocks (WebAssembly#1594)

* [interpreter] Run JS tests via node.js (WebAssembly#1595)

* [spec] Remove stray `x` indices (WebAssembly#1598)

* [spec] Style tweak for cross-refs

* [spec] Style eps (WebAssembly#1601)

* Separate subsumption from instr sequencing

* State principal types

* Add statements about glbs, lubs, and disjoint hierarchies

* Add missing bot

* [spec] Clarify that atoms can be symbolic (WebAssembly#1602)

* [test] Import v128 global (WebAssembly#1597)

* Update Overview.md

* [js-api] Expose everywhere

* [js-api] Try to clarify NaN/infinity handling. (WebAssembly#1535)

* [web-api] Correct MIME type check. (WebAssembly#1537)

Fixes WebAssembly#1138.

* [ci] Pin nodejs version to avoid fetching failures (WebAssembly#1603)

The issues appears to be related to actions/runner-images#7002.

Co-authored-by: Ms2ger <Ms2ger@igalia.com>

* [spec] Add missing value to table.grow reduction rule (WebAssembly#1607)

* [test] Move SIMD linking test to simd dir (WebAssembly#1610)

* Editorial: Clarify the name of the instantiate algorithm.

* Add notes to discourage using synchronous APIs.

* [jsapi] Normative: Always queue a task during asynchronous instantiation

JSC will have to do asynchronous compilation work during some instantiations.
To be consistent, this PR always queues a task to complete instantiation,
except through the synchronous Instance(module) API, to ensure consistency
across platforms.

This patch also cleans up the specification in various surrounding ways:
- Include notes about APIs whose use is discouraged/may be limited

Closes WebAssembly#741
See also webpack/webpack#6433

* [test] Exception -> Tag in wasm-module-builder.js

The section name has changed to the tag section a few years ago. This
adds the corresponding changes added in
WebAssembly/exception-handling#252 and
WebAssembly/exception-handling#256.

* [spec] Fix reduction rule for label (WebAssembly#1612)

Fix WebAssembly#1605.

* [spec] Clarifying note about canonical NaNs (WebAssembly#1614)

* [spec] Tweak crossref

* [test] Fix invalid section ID tests (WebAssembly#1615)

* [tests] Disable node run for now

* [spec] Don't check in generated index, to avoid spurious merge conflicts

* [spec] Rename script

* [ci] deactivate node run for now

* Fix uses of \to; compositionality

* Fix typo in text expansion

* Follow-up fix

* Fix compilation errors after merge.

This commit fixes the errors introduced by the merge of
function-references/main into this tree.

---------

Signed-off-by: Adrian Cole <adrian@tetrate.io>
Co-authored-by: Andreas Rossberg <rossberg@dfinity.org>
Co-authored-by: Hans Höglund <hanshoglund@users.noreply.github.com>
Co-authored-by: Ng Zhi An <zhin@chromium.org>
Co-authored-by: Ng Zhi An <zhin@google.com>
Co-authored-by: Sam Clegg <sbc@chromium.org>
Co-authored-by: Thomas Lively <7121787+tlively@users.noreply.github.com>
Co-authored-by: Gabor Greif <ggreif@gmail.com>
Co-authored-by: Andreas Rossberg <rossberg@mpi-sws.org>
Co-authored-by: Crypt Keeper <64215+codefromthecrypt@users.noreply.github.com>
Co-authored-by: Andreas Rossberg <rossberg@chromium.org>
Co-authored-by: Ng Zhi An <ngzhian@gmail.com>
Co-authored-by: Ben L. Titzer <ben.titzer@gmail.com>
Co-authored-by: Keith Winstein <keithw@cs.stanford.edu>
Co-authored-by: gahaas <ahaas@google.com>
Co-authored-by: r00ster <r00ster91@protonmail.com>
Co-authored-by: Timothy McCallum <tpmccallum@users.noreply.github.com>
Co-authored-by: Julien Cretin <github@ia0.eu>
Co-authored-by: Julien Cretin <cretin@google.com>
Co-authored-by: Ole Krüger <ole@vprsm.de>
Co-authored-by: Jämes Ménétrey <james@menetrey.me>
Co-authored-by: Ivan Panchenko <39594356+ivan-pan@users.noreply.github.com>
Co-authored-by: Ethan Jones <ictrobot@outlook.com>
Co-authored-by: Ole Krüger <ole.kruger@trili.tech>
Co-authored-by: aathan <aathan_github@memeplex.com>
Co-authored-by: Alberto Fiori <9143617+fifofefe@users.noreply.github.com>
Co-authored-by: mnordine <marknordine@gmail.com>
Co-authored-by: cosine <CosineP@users.noreply.github.com>
Co-authored-by: ariez-xyz <41232910+ariez-xyz@users.noreply.github.com>
Co-authored-by: Surma <surma@surma.dev>
Co-authored-by: Asumu Takikawa <asumu@igalia.com>
Co-authored-by: Ian Henderson <ian@ianhenderson.org>
Co-authored-by: Tom Stuart <hi@tomstu.art>
Co-authored-by: James Browning <thejamesernator@gmail.com>
Co-authored-by: whirlicote <78504913+whirlicote@users.noreply.github.com>
Co-authored-by: Ms2ger <Ms2ger@igalia.com>
Co-authored-by: Adam Lancaster <Adzz@users.noreply.github.com>
Co-authored-by: Ömer Sinan Ağacan <omeragacan@gmail.com>
Co-authored-by: B Szilvasy <beatrice.szilvasy@gmail.com>
Co-authored-by: Thomas Lively <tlively123@gmail.com>
Co-authored-by: Michael Ficarra <github@michael.ficarra.me>
Co-authored-by: YAMAMOTO Takashi <yamamoto@midokura.com>
Co-authored-by: Thomas Lively <tlively@google.com>
Co-authored-by: candymate <31069474+candymate@users.noreply.github.com>
Co-authored-by: Bongjun Jang <bongjun.jang@kaist.ac.kr>
Co-authored-by: ShinWonho <50018375+ShinWonho@users.noreply.github.com>
Co-authored-by: 서동휘 <hwidongsuh@gmail.com>
Co-authored-by: Jim Blandy <jimb@red-bean.com>
Co-authored-by: Heejin Ahn <aheejin@gmail.com>
Co-authored-by: Daniel Ehrenberg <littledan@chromium.org>
backes pushed a commit to backes/spec that referenced this pull request Jul 12, 2023
@rossberg
[spec] Tweak tokenisation for text format (WebAssembly#1499)
bce1aae
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@tlively tlively tlively approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@rossberg @tlively