Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide a toml module in the standard library #84240

Closed
mgorny mannequin opened this issue Mar 25, 2020 · 27 comments
Closed

Provide a toml module in the standard library #84240

mgorny mannequin opened this issue Mar 25, 2020 · 27 comments
Labels
stdlib Python modules in the Lib dir type-feature A feature request or enhancement

Comments

@mgorny
Copy link
Mannequin

mgorny mannequin commented Mar 25, 2020

BPO 40059
Nosy @brettcannon, @tiran, @mcepl, @njsmith, @encukou, @agronholm, @mgorny, @dstufft, @pradyunsg, @eli-schwartz, @miss-islington, @tirkarthi, @skoslowski, @erlend-aasland, @hauntsaninja, @domdfcoding, @hukkin, @YakoYakoYokuYoku
PRs
  • bpo-40059: tomllib #31498
  • bpo-40059: Fix installation of tomllib #31784
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = None
    created_at = <Date 2020-03-25.06:54:21.437>
    labels = ['type-feature', 'library']
    title = 'Provide a toml module in the standard library'
    updated_at = <Date 2022-03-09.13:38:15.300>
    user = 'https://github.com/mgorny'

    bugs.python.org fields:

    activity = <Date 2022-03-09.13:38:15.300>
    actor = 'miss-islington'
    assignee = 'none'
    closed = False
    closed_date = None
    closer = None
    components = ['Library (Lib)']
    creation = <Date 2020-03-25.06:54:21.437>
    creator = 'mgorny'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 40059
    keywords = ['patch']
    message_count = 27.0
    messages = ['364979', '364983', '365000', '373879', '387807', '387892', '401071', '401421', '406458', '406484', '406488', '406515', '406561', '408316', '408320', '408418', '408503', '408505', '408576', '409488', '409489', '409552', '410353', '414731', '414746', '414800', '414801']
    nosy_count = 20.0
    nosy_names = ['brett.cannon', 'christian.heimes', 'mcepl', 'njs', 'petr.viktorin', 'alex.gronholm', 'mgorny', 'dstufft', 'pradyunsg', 'eschwartz', 'miss-islington', 'VA', 'xtreak', 'skoslowski', 'erlendaasland', 'hauntsaninja', 'domdfcoding', 'dom1310df', 'hukkinj1', 'YakoYakoYokuYoku']
    pr_nums = ['31498', '31784']
    priority = 'normal'
    resolution = None
    stage = 'patch review'
    status = 'open'
    superseder = None
    type = 'enhancement'
    url = 'https://bugs.python.org/issue40059'
    versions = []

    @mgorny
    Copy link
    Mannequin Author

    mgorny mannequin commented Mar 25, 2020

    PEP-518 uses the TOML format to specify build system requirements. AFAIU this means that all new build systems will require a TOML parser. Could you consider adding one to the standard library to reduce the number of chicken-egg problems?

    The referenced PEP states that 'pytoml TOML parser is ~300 lines of pure Python code', so I don't think integrating it would be a large maintenance cost.

    [1] https://www.python.org/dev/peps/pep-0518/

    @mgorny mgorny mannequin added 3.9 stdlib Python modules in the Lib dir type-feature A feature request or enhancement labels Mar 25, 2020
    @tirkarthi
    Copy link
    Member

    tirkarthi commented Mar 25, 2020

    Relevant python-dev discussion : https://mail.python.org/pipermail/python-dev/2019-May/157405.html . The format has still not reached 1.0. Issue to track the 1.0 release candidate : toml-lang/toml#698

    @brettcannon
    Copy link
    Member

    brettcannon commented Mar 25, 2020

    The plan is to start discussing adding a TOML parser once the spec reaches 1.0 (which we will know about as one of the pip contributors also manages TOML).

    @VA
    Copy link
    Mannequin

    VA mannequin commented Jul 18, 2020

    1.0.0-rc.1 is out by now: https://github.com/toml-lang/toml/blob/master/CHANGELOG.md

    @VA VA mannequin added 3.10 labels Jul 18, 2020
    @mgorny
    Copy link
    Mannequin Author

    mgorny mannequin commented Feb 28, 2021

    It seems that the spec has reached 1.0.0:

    1.0.0 / 2021-01-11

    I guess that there are still ~2 months left to do this before 3.10 reaches beta.

    @brettcannon
    Copy link
    Member

    brettcannon commented Mar 2, 2021

    We will see if anyone has enough time to get this done between now and May for 3.10, but discussions have been started with some folks to propose something for the stdlib.

    @mgorny
    Copy link
    Mannequin Author

    mgorny mannequin commented Sep 5, 2021

    Is there any progress happening? FWIU the common "toml" module on pypi has been discontinued now, projects are switching to yet another one and this is exactly the kind of problem a built-in module would have avoided.

    @brettcannon
    Copy link
    Member

    brettcannon commented Sep 8, 2021

    No progress as I've been swamped with higher-priority things and the bigger discussion about how we want to manage the stdlib going forward has not started yet (once again, not had the time to start that).

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Nov 17, 2021

    I'd love it if we could make this happen for Python 3.11. What can I do to help, e.g. would it be helpful if I drafted a PEP?

    Several projects have switched to using tomli (https://github.com/hukkin/tomli), which is fully compatible with TOML v1.0.0 and is maybe 800 LoC with claimed 100% branch coverage.

    There's also tomlkit, which has been around for longer, but is perhaps larger and more feature-ful than we'd want in stdlib, e.g. it allows you to roundtrip dump and load TOML documents preserving comments, whitespace, etc.

    @brettcannon
    Copy link
    Member

    brettcannon commented Nov 17, 2021

    I'd love it if we could make this happen for Python 3.11. What can I do to help, e.g. would it be helpful if I drafted a PEP?

    A PEP isn't necessarily required right now as https://discuss.python.org/t/how-do-we-want-to-manage-additions-removals-to-the-stdlib/10681 needs to be resolved by the steering council first (it's on our agenda).

    Several projects have switched to using tomli (https://github.com/hukkin/tomli), which is fully compatible with TOML v1.0.0 and is maybe 800 LoC with claimed 100% branch coverage.

    Part of the issue with tomli is how new it is; https://pypi.org/project/tomli/#history shows it is only 6 months old and we typically require a project be at least a year old before we consider pulling it in.

    If you want to help you could ask the author of tomli what they think about it being added to the stdlib and whether they support that idea.

    @mgorny
    Copy link
    Mannequin Author

    mgorny mannequin commented Nov 17, 2021

    I've already asked that, and the author said it's fine but with the deprecated text file support removed:

    hukkin/tomli#141 (comment)

    That said, for consistency with json and so on we'd probably want to combine tomli and tomli_w to get dumping support as well.

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Nov 18, 2021

    Hopefully tomli being less than a year old should not be a blocker, after all, TOML v1.0.0 is itself less than a year old.

    Despite tomli being new code, it's pretty widely used. Reverse dependencies include pip, pytest, mypy, black, flit, coverage, setuptools-scm, cibuildwheel. All in all, it should be well proven in practice by the time it comes to make 3.11 stable.

    It also helps that it's easy to anticipate the interface: dump, dumps, load, loads. Overall, hopefully risk here should be well within the budget of a provisional module.

    That's all to say, if and when the Steering Council decides that there's work to be done here, let me know how I can help :-)

    @brettcannon
    Copy link
    Member

    brettcannon commented Nov 18, 2021

    Hopefully tomli being less than a year old should not be a blocker, after all, TOML v1.0.0 is itself less than a year old.

    It's actually rather important as the semantics of how things work will effectively be locked in once a module is added to the stdlib, so we need to be careful. Plus had you asked this question less than a year go the answer would have been pypi.org/p/toml and now you're suggesting something different, so the year matters. 😄

    @YakoYakoYokuYoku
    Copy link
    Mannequin

    YakoYakoYokuYoku mannequin commented Dec 11, 2021

    A new python-ideas mail thread was created for this, you can check it out at https://mail.python.org/archives/list/python-ideas@python.org/thread/IWJ3I32A4TY6CIVQ6ONPEBPWP4TOV2V7/.

    @tiran
    Copy link
    Member

    tiran commented Dec 11, 2021

    I just noticed that tomli has dropped support for Python 3.6. That's a road block for general adoption of the package in the Python ecosystem. Python 3.6 is the default Python interpreter in CentOS 8, C8S, RHEL 8, and Ubuntu 18.04 LTS. hukkin/tomli#134

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Dec 13, 2021

    Given that this currently seems blocked on the broad question of "how should additions and removals to the stdlib be managed", I'd like to not focus too hard just yet on the specifics of tomli. I assume it's unlikely, but for all we know, the SC could determine that all newly included modules have to be written from scratch, following along the lines of recent additions like zoneinfo, graphlib and importlib.metadata.

    There's a lot we could bikeshed or debate... e.g., it's not even clear to me what a toml package in the stdlib would be named, never mind what it means for an unreleased version of a commonly used third party toml package dropping support for an imminently EOL version of Python. I'm possibly totally out of line, but in my head, the process would look something like this:

    Step 1: Hear back from the SC about criteria and considerations for new modules in the stdlib
    Step 2: Determine whether a TOML library *in the abstract* is something that would meet the outlined criteria (potentially e.g. is this something we even want? is this something we can maintain?)
    Step 3: Determine if we have an implementation (written from scratch, copied, or derived from something pre-existing) that would meet the outlined criteria
    Step 4: Do all the rest of the work to meet the outlined criteria (potentially e.g. go through PEP process, create proposed impl, write a backport, bikeshed api and name, etc)

    I guess I have the following questions:

    • Is my understanding correct that this issue is blocked on SC guidance?
    • Is there anything we could do in advance of SC guidance that would be productive?
      Brett previously mentioned bringing it up with the author of tomli, as per Please consider pushing tomli into stdlib hukkin/tomli#141 they seem supportive
    • Is there a good place to follow along or be notified of SC thoughts?
      I see no mention of stdlib changes in https://github.com/python/steering-council and the discuss thread linked above seems to have petered out.

    @brettcannon
    Copy link
    Member

    brettcannon commented Dec 14, 2021

    I just noticed that tomli has dropped support for Python 3.6. That's a road block for general adoption of the package in the Python ecosystem.

    It's already in pip, so I think it's already generally adopted 😉. https://github.com/pypa/pip/tree/main/src/pip/_vendor/tomli

    Is my understanding correct that this issue is blocked on SC guidance?

    Not officially, no. But I'm personally not going to bring it forward right now. If someone else wants to formulate a complete proposal for the SC on this then they are definitely welcome to! You will need to address where the code is coming from, why that code should be used, what's the API, etc.

    The only reason the SC is mentioned here is there will be a discussion about how to maintain the stdlib, but it simply hasn't happened yet. You don't have to wait for it and asking for a TOML module might actually force the issue.

    Is there anything we could do in advance of SC guidance that would be productive?

    Nope, someone eventually has to have the time to make the proposal and manage the deluge of comments.

    Is there a good place to follow along or be notified of SC thoughts?

    https://github.com/python/steering-council as you already pointed out through the issues and monthly summaries. Otherwise you just need to open an issue and ask. 😃

    @YakoYakoYokuYoku
    Copy link
    Mannequin

    YakoYakoYokuYoku mannequin commented Dec 14, 2021

    Not officially, no. But I'm personally not going to bring it forward right now. If someone else wants to formulate a complete proposal for the SC on this then they are definitely welcome to! You will need to address where the code is coming from, why that code should be used, what's the API, etc.

    The only reason the SC is mentioned here is there will be a discussion about how to maintain the stdlib, but it simply hasn't happened yet. You don't have to wait for it and asking for a TOML module might actually force the issue.

    I've been working on a document detailing the API implementation (as a PEP draft) see https://gitlab.com/YakoYakoYokuYoku/pep-toml. Although I've yet to write the code I'll to move forward with it.

    @brettcannon
    Copy link
    Member

    brettcannon commented Dec 14, 2021

    I opened python/steering-council#92 for the SC to discuss stdlib additions in case I am not re-elected.

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Jan 2, 2022

    You will need to address where the code is coming from, why that code should be used, what's the API, etc.

    Happy new year, potentially-toml-wanting friends!

    I wrote up a draft of a proposal here: https://gist.github.com/hauntsaninja/9f136a5a60f63d8ca2cdfadb50edba44

    I've passed it along to hukkin (author of tomli) for feedback. Once they reply, I'll send it to python-dev and the deluge of comments can get underway. If in the meantime anyone else has thoughts, I'd love to hear from you.

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Jan 2, 2022

    pradyunsg kindly pointed me to an ongoing thread in Packaging discuss: https://discuss.python.org/t/adopting-recommending-a-toml-parser/4068/84

    So seems like a PEP will be necessary. I'm happy to do legwork for that. (The PEP that I'll write differs from YakoYakoYokuYoku's json-esque draft in that it'll be based on tomli instead of a from-scratch implementation)

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Jan 3, 2022

    We've started a PEP draft.

    https://github.com/hauntsaninja/peps/blob/toml-pep/pep-9999.rst shows a rendered version.
    hauntsaninja/peps#1 is a PR from the toml-pep branch to main, to help ease of review / discussion.

    If you're following along and need to get your bearings straight, here are the recently active locations of discussion:

    @hauntsaninja
    Copy link
    Contributor

    hauntsaninja commented Jan 11, 2022

    @encukou
    Copy link
    Member

    encukou commented Mar 8, 2022

    New changeset 591f675 by Taneli Hukkinen in branch 'main':
    bpo-40059: Add tomllib (PEP-680) (GH-31498)
    591f675

    @encukou
    Copy link
    Member

    encukou commented Mar 8, 2022

    The PR is merged and buildbots are green. Thank you to everyone who helped!

    Now would be a good time to bikeshed wording in the documentation.

    From the PR:

    Would it be good to mention in the docs why load() takes only binary files? The encoding requirement probably isn't obvious for first-time users.

    @dom1310df
    Copy link
    Mannequin

    dom1310df mannequin commented Mar 9, 2022

    When building Python from source (as of the latest GitHub commit) the tomllib directory doesn't actually get copied over to the install prefix.

    It looks like an entry's needed in Makefile.pre.in under LIBSUBDIRS, along the lines of #13563 and #30311

    @miss-islington
    Copy link
    Contributor

    miss-islington commented Mar 9, 2022

    New changeset 23dcea5 by Dominic Davis-Foster in branch 'main':
    bpo-40059: Fix installation of tomllib (GH-31784)
    23dcea5

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    @encukou encukou closed this as completed Jun 27, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    stdlib Python modules in the Lib dir type-feature A feature request or enhancement
    Projects
    None yet
    Development

    No branches or pull requests

    6 participants