-
Notifications
You must be signed in to change notification settings - Fork 847
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make toml-test
the standard test suite for checking compliance of implementations
#585
Comments
Not yet. |
Is there a doc on the diff between v0.4 and v0.5? |
There is a changelog which lists what has changed. |
If your goal is to test an existing parser I suggest looking at: BurntSushi's test suite: https://github.com/BurntSushi/toml-test And mine: https://github.com/iarna/toml-spec-tests (Which is in a compatible file format but independently created and differs mostly by having more complete support for TOML 0.5) I also maintain a supported-feature chart for the various TOML parsers on npm that's produced by running them through these two test suites: TOML-SPEC-SUPPORT |
Thanks!
…On Thu, Jan 17, 2019, 6:45 PM Rebecca Turner ***@***.*** wrote:
If your goal is to test an existing parser I suggest looking at:
BurntSushi's test suite: https://github.com/BurntSushi/toml-test
And mine: https://github.com/iarna/toml-spec-tests
(Which is in a compatible file format but independently created and
differs mostly by having more complete support for TOML 0.5)
I also maintain a supported-feature chart for the various TOML parsers on
npm that's produced by running them through these two test suites.
TOML-SPEC-SUPPORT
<https://shared.by.re-becca.org/misc/TOML-SPEC-SUPPORT.html>
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#585 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAI4lrtK9vPLCZh120LIAF8XNSBr1XsNks5vETVNgaJpZM4Z84vv>
.
|
@iarna 👋🏻 Would you have any pointers on what the current state of affairs is on this front, with 0.5 compliance testing? I'm trying to figure out what "blessed" compliance tests for TOML 1.0 should look like - they'll live in https://github.com/toml-lang/compliance. I expect/hope we can reuse @BurntSushi and @iarna's work for this. I am agnostic to the language/tech we use for this, as long as we document how to get the compliance tests running for parser implementers and it's not "too difficult". :) |
Heya. Mine should be complete for 0.5 these days. As an implementer, most of what I want is paired files: "here's some toml" "here's what it should represent". Most of my tests are toml/yaml pairs, where the yaml should produce the same document. There are a couple of mine that follow @BurntSushi's lead and use json, with named data types, and stringified representations, when testing for types that don't exist in yaml. I chose yaml for two resons: First, it is how the existing tests folder in the toml repo is setup. Second, it was easier to compose while filling out the test suite. That said, having everything in JSON format as BurntSushi did would be more consistent (only one file type) and save implementers from having to find a yaml implementation. It'd be pretty easy to mechanically convert the yaml files to the JSON format. |
What does Standard Test Suite refer to?
|
I am using both @BurntSushi and @iarna tests in my parser implementation. One thing I like about @iarna is that there are some qa-* tests defined, which really pushed me into improving the processing speed of my parser. I am not sure if they would belong in a blessed test set, provided directly from the TOML repo, but it would be nice to have them available as a secondary performance test set. |
Anyone who wants to add to the test suite, or add more qa-type tests, I'm happy to add as a contributor. (Or even move it to a shared org.) |
Making this a critical path item for TOML 1.0, since it makes a lot of sense to do a compliance test suite, in time for TOML 1.0. |
Since @cannikin is handling the website, I'm going be working on this. 🙃 |
Stating what probably everyone already knows: The https://github.com/BurntSushi/toml-test DSL built on top of JSON looks good as a baseline, since it relies on returning a JSON object with a name, stringified value, and type annotation, and anything that can do JSON can do that, and since it is abstract (it doesn't really bind itself to fundamental JSON qualities, aside from being able to represent a hashmap, and uh, strings), it doesn't seem to have any obvious drawbacks aside from "also you have to parse JSON", which is a pretty low bar to clear. @iarna's logic for YAML : TOML comparisons being easy is compelling but also hits the wall mentioned. The harness itself using Go is secondary, and he does link to a few other projects which replicate the harness. I'm a little uncertain how one would go about extracting the test suite from https://github.com/iarna/iarna-toml as a freestanding test suite that could be called by e.g. a Bash script, but it also is quite comprehensive, and probably is the most up to date and thorough set of tests, barring the recent edits. I am guessing you would probably prefer to wrangle a test suite in Python, however, on the code side, and I mean, it's kind of hard to go wrong with "most popularly used language", if we have to offer a piece of software to use, as long as it has no dependencies. Whatever it is, the format probably should include code-less data files and instructions on how to (re)build the harness. |
I'm up for converting my own test suite to the same format as the BurntSushi one, at which point there's no integration issues, we can use the go implementation or any other. (This can be done mechnically: They're currently just ymal files representing the object we expect, they can be easily converted to the json format.) Edited to add: |
Ah that would explain it, I somehow missed an entire repo! Good job, me. |
As far as I can see, this seems to be the only still open issue that blocks the release of TOML 1.0? Hope it won't block too long... (The world has waited long enough, me thinks.) |
Besides the tests themselves, there needs to be an specification of the toml -> json encoding rules. If I'm not wrong, @iarna 's tests follow BurntSushi's guidelines, which would need to be updated since they are missing local datetime, date and time types. (I addressed this in this P.R. toml-lang/toml-test#51 , using the same conventions as Iarna's, if I remember well.) |
One more reason to get an official location for test cases, however imperfect it may be at first, is for those that document updates to the spec. Case in point: I created #728 to clarify something in the spec and wanted to include a test case with the associated PR. I'd like to add it to https://github.com/toml-lang/compliance, but that repo is completely empty. I also don't see a test for this case in either https://github.com/BurntSushi/toml-test/ or https://github.com/iarna/toml-spec-tests/, but neither has been updated in over a year (but maybe that'll change soon with the v1.0.0-rc.1 being out), so I worry that my test case would be forgotten if I submitted a PR to those. There's also the I guess the best I can do for now is add the test to this issue thread? If so, here it is: When the last non-whitespace character in a multi-line string is a backslash it ignores any following whitespace, but not if the backslash is escaped. A naive parser might just check if the last non-whitespace character is ml-escaped-nl = """
foo \
bar \\
baz \\\
quux""" |
Your PR wouldn't be forgotten, but I too would prefer that compliance tests consolidate in one (official) place. |
I've added a branch to my https://github.com/iarna/toml-spec-tests/tree/1.0.0-rc.1 This is updated from the diff to the English docs between 0.5.0 and 1.0.0-rc.1. Likely it could use a second pass with an eye toward bringing in details from the ABNF. |
Now that's just splitting hairs. The date format is strictly ISO8601, the canonical format for this kind of data. All I'm asking for is that floats get a similarly well-defined format (like exponential notation with 6 digits or something like that) so that I don't have to special case them. |
I'm really not splitting hairs: ISO8601 doesn't have a singular string representation for a given date. There are many valid representations of the same date. For ex |
I updated https://github.com/BurntSushi/toml-test with a bunch of additional tests; I think it should support most of 1.0 now, but I need to go over some details and there's probably a few things I missed. I also added a bunch of unit tests in https://github.com/BurntSushi/toml that I need to port over, and there are a few further improvements to the test runner tool that can be made. Either way, after some period of inactivity it's maintained now (by me). Just FYI. |
What's the path forward on this? There are now essentially two "competing" solutions: BurntSushi/toml-test and toml/compliance When I started working on this a few weeks ago toml/compliance seemed like an empty placeholder repo as it didn't contain any actual tests, but since then someone opened a PR to actually add some tests, and looking at run.py a bit closer it does actually implement a test runner. Practically speaking:
Personally, I think it makes sense to replace toml/compliance with BurntSushi/toml-test. BurntSushi already said he's fine with that, and I'm fine with it too. I mean, people could spend a lot of effort on fixing all of this in toml/compliance, but it seems a bit pointless if there's already a good solution? Especially since this is a fairly niche tool where you don't really need 2 or 3 different solutions that people can choose from according to preferences/use case. |
And I agree, especially given that toml-test is actively maintained now. :) The only concern I have is the mechanics of making this happened -- I don't have the admin bit on the |
@pradyunsg What do you need from me? Happy to go along with whatever y'all decide. |
With toml-lang/compliance#11, I've pointed toml/compliance to BurntSushi/toml-test. @BurntSushi @mojombo I think we should transfer the |
toml-test
the standard test suite for checking compliance of implementations
Filed an issue over at toml-test, for the maintainers to chime in. We'll discuss the logistics there, and I'll close this issue out once the transfer is done and the spec is updated to reflect the changes to be made. |
It's now moved to https://github.com/toml-lang/toml-test I got the same kind of collaborator/commit access as before, so no further action is needed for that. We will probably want to delete https://github.com/toml-lang/compliance? Could also archive it, but not too much value in keeping it, I think? |
@pradyunsg I think we can close this? Or do we need any further action? |
We're done here. Thanks for the patience and the continued maintenance of toml-test! |
No description provided.
The text was updated successfully, but these errors were encountered: