Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PEP 750: Tag Strings For Writing Domain-Specific Languages #3858

Merged
merged 42 commits into from
Aug 9, 2024

Conversation

jimbaker
Copy link
Contributor

@jimbaker jimbaker commented Jul 8, 2024

This PEP introduces tag strings for custom, repeatable string processing. Tag strings are an extension to f-strings, with a custom function -- the "tag" -- in place of the f prefix. This function can then provide rich features such as safety checks, lazy evaluation, domain specific languages (DSLs) for web templating, and more.

Tag strings are similar to JavaScript tagged template literals and related ideas in other languages.


📚 Documentation preview 📚: https://pep-previews--3858.org.readthedocs.build/

@jimbaker jimbaker requested a review from a team as a code owner July 8, 2024 23:12
Copy link

cpython-cla-bot bot commented Jul 8, 2024

All commit authors signed the Contributor License Agreement.
CLA signed

.github/CODEOWNERS Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
jimbaker and others added 17 commits July 9, 2024 16:22
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Lambda-wrapped expressions use class scope
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
@Carreau
Copy link
Contributor

Carreau commented Jul 25, 2024

This is great. I hope we'll get log.debug"", log.info"" and alike. I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

@jimbaker
Copy link
Contributor Author

This is great. I hope we'll get log.debug"", log.info"" and alike.

In this PEP, we don't support dotted names as prefixes for tag strings. We looked into it, and it required a fairly large number of parser changes. We decided to keep it simple instead. But dotted names should be supportable in a future PEP, if there's interest.

I've so many project use fstrings in logging, which prevents doing proper structured logging later. Though I do have the impression that tags having side-effects is not mentioned in the pep (beyond memoizing), but most example and discussions are around pure tags I think.

So the way I would think we would support logging integration is as follows, and it would enable structured logging:

Tag function. Create a tag function struct_log (it could be imported with a short name like sl). It has a signature like so:

def struct_log(*args: str | Interpolation) -> LogRecordMessage:
    ...

where LogRecordMessage holds the strings and interpolations for evaluation, if and when written to a log. It should be a simple wrapper, and is used to support mixed usage with the usual strings and args passed into log.info, etc.

@dataclass
class LogRecordMessage:
    args: tuple[str | Interpolation, ...]

A few things:

  • Logging most likely doesn't need Decoded.raw, so str can be used.
  • The specific tag function could be built by some sort of configurator. Example: provide a tag function that expands all usages of {expr} to {expr=value}.
  • Likewise, this can be configured such that structured logging can use Interpolation.expr when writing out a JSON structured log.
  • Another possible configuration would be Interpolation.getvalue when evaluated returns a Pydantic model, it can use that library's support for serializing to JSON. Lots of possibilities.

Now add the following to the above class:

    def __str__(self):
         # evaluate as if a lazy f-string

With __str__, this provides instant integration with logging given that LogRecord.getMessage calls str on the message.

Logging integration. The most straightforward way to do this is with setLogRecordFactory on a logger; this factory will be a subclass of LogRecord; it will have its own getMessage. This then works with a subclass of Formatter to build the final text representation, whether that's output of the above as if it were a f-string; or a JSON structured log version. I haven't used this library, but https://github.com/madzak/python-json-logger is one possible example for supporting such structured logging. There are also examples in the Logging Cookbook.

I'm also guessing projects like mypy will automatically detect (*args: Decoded | Interpolation) -> Any as potential tags.

That's an interesting idea. Definitely it goes the opposite way.

@jimbaker
Copy link
Contributor Author

This PEP should be ready for approval, now that all earlier questions have been addressed.
@python/pep-editors

peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
peps/pep-0750.rst Outdated Show resolved Hide resolved
Copy link
Contributor

@willingc willingc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, I like this PEP a lot. I've made some suggestions to help with clarity and distinction from f-strings.

pauleveritt and others added 7 commits August 2, 2024 08:16
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
* Include three comments from @Carreau (drop two bullets, *args in example, explain *args.)

* Lambda-wrapped expressions use annotation scope

* Clarify use of annotation scope

* Mention what happens to named unicodes followed by text

* Use DecodedConcrete in assertion

* Rewrite why annotation scope is needed (#4)

* Rewrite why annotation scope is needed

* Minor copyediting

* PEP 747: Fix rules related to UnionType (T1 | T2). Contrast TypeExpr with TypeAlias. Apply other feedback. (python#3856)

* PEP 694: Fix typo (python#3859)

* PEP 2026: Update following discussion (python#3860)

Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>

* PEP 101: Remove outdated info and add new info (python#3863)

* PEP 101: Remove outdated info

* PEP 101: Update make command for running tests

* PEP 101: Replace '#python-dev and/or python-committers' with 'Discord and/or Discourse

* PEP 101: Add Hugo as 3.14 RM

* PEP 101: Add to PSRT

* PEP 11: Add Russell as an iOS contact (python#3865)

* Meta: Document the PEPs API (python#3864)

Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>

* PEP 719: Update for today's release of 3.13.0b4 (python#3868)

* PEP 740: Mark as Provisional (python#3848)

Signed-off-by: William Woodruff <william@yossarian.net>

* PEP 749: Add section on metaclasses (python#3847)

Co-authored-by: Carl Meyer <carl@oddbird.net>

* PEP 8: Update a Wikipedia link (python#3552)

* PEP 635: Minor typo fix in code sample (python#3871)

Looks like an unclosed f-string.

* PEP 751: A file format to list Python dependencies for installation reproducibility (python#3870)

Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>

* PEP 743: Rewrite to hide (soft-)deprecated API (pythonGH-3869)

Co-authored-by: Victor Stinner <vstinner@python.org>

* PEP 751: Add Discussions-To and Post-History (python#3872)

* PEP 639: Incorporate the latest discussion feedback (python#3866)

* Remove the requirement of license-files defaults

* Cover all rejected subkeysideas in one paragraph

* Change the deprecation policy around classifiers

* Flatten the value of the license-files key, only globs are specified

* Update the Rejected ideas to match the current license-files proposal

---------

Co-authored-by: Miro Hrončok <miro@hroncok.cz>

* PEP 715: clarify what `[package.tool]` is (python#3873)

* PEP 665: Superseded-By: 751 (python#3875)

* PEP 751: update based on feedback (python#3877)

* PEP 751: update based on feedback

* Fix a section underline

* Include three comments from @Carreau (drop two bullets, *args in example, explain *args.)

* From Carol, move the point about import to the following paragraph.

* Per Carol: Remove paragraph about lifecycles as that is about *a* DSL, not DSLs in general.

---------

Signed-off-by: William Woodruff <william@yossarian.net>
Co-authored-by: pauleveritt <paul.everitt@jetbrains.com>
Co-authored-by: Jim Baker <jim.baker@python.org>
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: David Foster <david@dafoster.net>
Co-authored-by: Barry Warsaw <barry@python.org>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Erlend E. Aasland <erlend.aasland@protonmail.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: T. Wouters <thomas@python.org>
Co-authored-by: William Woodruff <william@yossarian.net>
Co-authored-by: Jelle Zijlstra <jelle.zijlstra@gmail.com>
Co-authored-by: Carl Meyer <carl@oddbird.net>
Co-authored-by: Lavrentiy Rubtsov <rnbsov@gmail.com>
Co-authored-by: Mariatta <Mariatta@users.noreply.github.com>
Co-authored-by: Brett Cannon <brett@python.org>
Co-authored-by: Carol Willing <carolcode@willingconsulting.com>
Co-authored-by: Petr Viktorin <encukou@gmail.com>
Co-authored-by: Victor Stinner <vstinner@python.org>
Co-authored-by: Karolina Surma <33810531+befeleme@users.noreply.github.com>
Co-authored-by: Miro Hrončok <miro@hroncok.cz>
Copy link
Contributor

@willingc willingc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the changes.

@lysnikolaou
Copy link
Contributor

Thanks everyone for the reviews! I'll merge this to get the conversation started and we can always come back to fix more issues.

@lysnikolaou lysnikolaou merged commit b1d591f into python:main Aug 9, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants