Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Text/replace builtin #1065

Merged
merged 25 commits into from
Oct 10, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
25 commits
Select commit Hold shift + click to select a range
e083884
Add Text/replace builtin
Sep 6, 2020
42fb6ff
Update standard/beta-normalization.md
alexhumphreys Sep 7, 2020
db056f0
Add extra unicode test
Sep 7, 2020
df2405b
fix line formatting
Sep 7, 2020
bb814eb
Define behaviour by induction in standard/beta-normalization.md
alexhumphreys Sep 16, 2020
ae955b9
Merge branch 'master' into feature/text/replace
alexhumphreys Sep 16, 2020
9182692
Add a note about empty 'needle' argument
Sep 16, 2020
2577b6e
Add named arguments in docs/references/Built-in-types.md
alexhumphreys Sep 20, 2020
cae8ec7
Add replace newlines example
Sep 20, 2020
a61ff5f
Reorder Text/replace beta-normalization rules
Sep 20, 2020
595c8ca
Add test for replacing an empty string
Sep 20, 2020
e67a627
Specify normalisation form for comparison
Sep 26, 2020
8fb45f5
Add Unicode NFC Text/replace test
Sep 28, 2020
6add36b
Merge branch 'master' into feature/text/replace
Gabriella439 Oct 3, 2020
e15215b
Update binary encoding for builtins test
Gabriella439 Oct 3, 2020
bd2948c
Update standard type for `Text/replace`
Gabriella439 Oct 3, 2020
008713a
Update the β-normalization rules
Gabriella439 Oct 3, 2020
87885ce
Add unit test for handling abstract arguments
Gabriella439 Oct 3, 2020
905f6b1
Don't handle Unicode normalization for now
Gabriella439 Oct 3, 2020
8affa98
Add missing decoding judgment
Gabriella439 Oct 3, 2020
2e9c134
Merge pull request #1 from dhall-lang/gabriel/Text/replace
alexhumphreys Oct 4, 2020
e217c2a
Swap subset for substring
Oct 4, 2020
3efd28f
Remove specific Unicode normalisation form
Oct 4, 2020
c03ffcf
Update docs/references/Built-in-types.md
alexhumphreys Oct 9, 2020
4d9d42f
Merge branch 'master' into feature/text/replace
alexhumphreys Oct 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions Prelude/Text/replace.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
{-
Replace a section of `Text` with another inside a `Text` literal.
-}
let replace
: ∀(needle : Text) → ∀(replacement : Text) → ∀(haystack : Text) → Text
= Text/replace

let example0 = assert : replace "-" "_" "foo-bar" ≡ "foo_bar"

let example1 = assert : replace "💣" "💥" "💣💣💣" ≡ "💥💥💥"
Gabriella439 marked this conversation as resolved.
Show resolved Hide resolved

let example2 = assert : replace "👨" "👩" "👨‍👩‍👧‍👦" ≡ "👩‍👩‍👧‍👦"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also add this as one of the test cases in the standard test suite?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, updated. Should I leave this in the prelude? Just wondering how easy this will be for all implementations, since most users importing the prelude won't need this particular replacement.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is also worth keeping in the Prelude since this is useful documentation for the end user. These examples will also show up in the generated docs


in replace
2 changes: 2 additions & 0 deletions docs/howtos/Cheatsheet.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@
"ABC" ++ "DEF" = "ABCDEF"

Text/show "Hello, world!" = "\"Hello, world!\""

Text/replace "-" "_" "foo-bar" = "foo_bar"
```

## Complex types
Expand Down
17 changes: 17 additions & 0 deletions docs/references/Built-in-types.md
Original file line number Diff line number Diff line change
Expand Up @@ -815,6 +815,23 @@ string:
"\"\\n🎉\""
```

### Function `Text/replace`

The `Text/replace` built-in function modifies a substring of a given `Text` literal. It takes 3 arguments, the `Text` literal substring to match, the `Text` literal replacement, and the `Text` literal in which to replace all matches:

```dhall
⊢ Text/replace "foo" "bar" "foobar"

"barbar"
```

#### Type

```
──────────────────────────────────────
Γ ⊢ Text/replace : ∀(needle : Text) → ∀(replacement : Text) → ∀(haystack : Text) → Text
```

#### Type

```
Expand Down
28 changes: 28 additions & 0 deletions docs/tutorials/Language-Tour.md
Original file line number Diff line number Diff line change
Expand Up @@ -910,6 +910,34 @@ You can concatenate `Text` literals using the `++` operator:
"123456"
```

It is also possible to replace substrings inside a Text literal. Say you have
a name where you need to replace dashes with underscores. That can be achieved
with the following:

```dhall
⊢ Text/replace "-" "_" "foo-bar-baz"

"foo_bar_baz"
```

You can also replace larger sections of text:
Gabriella439 marked this conversation as resolved.
Show resolved Hide resolved

```dhall
⊢ Text/replace "Hey" "Hello" "Hey, world!"

"Hello, world!"
```

Here's how you would strip newlines from a multiline string:

```dhall
Text/replace "\n" " " ''
This now behaves like a YAML
"folded" multi-line string literal where
newlines are replaced with spaces
''
```

Other than that, `Text` literals are essentially opaque. You currently cannot
parse `Text` literals nor can you compare them for equality. This is because
the language promotes using more precise types (like enums) instead of `Text`
Expand Down
1 change: 1 addition & 0 deletions standard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,7 @@ a, b, f, l, r, e, t, u, A, B, E, T, U, c, i, o
/ List/indexed ; Tag elements with index
/ List/reverse ; Reverse list
/ Text/show ; Convert Text to its own representation
/ Text/replace ; Replace a section of a Text literal
/ Bool ; Bool type
/ Optional ; Optional type
/ Natural ; Natural type
Expand Down
4 changes: 4 additions & 0 deletions standard/alpha-normalization.md
Original file line number Diff line number Diff line change
Expand Up @@ -422,6 +422,10 @@ sub-expressions for the remaining rules:
Text/show ↦ Text/show


───────────────────────────
Text/replace ↦ Text/replace


───────────
Bool ↦ Bool

Expand Down
45 changes: 43 additions & 2 deletions standard/beta-normalization.md
Original file line number Diff line number Diff line change
Expand Up @@ -661,13 +661,54 @@ Or in other words:
f a ⇥ "\"…\\n…\\u0024…\\\\…\\\"…\\u0000…\""


Otherwise, in isolation `Text/show` is in normal form:
`Text/replace` modifies a substring of a given `Text` literal. It takes 3
arguments, the `Text` literal substring to match, the `Text` literal replacement,
and the `Text` literal in which to replace all matches. In the case that the
substring to replace is empty (`""`), then no replacement is performed:


f ⇥ Text/replace "" replacement a ⇥ "foo" ; No replacement performed if
──────────────────────────────────────────── ; the search string is empty,
f a ⇥ "foo" ; to avoid an infinite loop.


f ⇥ Text/replace "NEEDLE" replacement ; If the "NEEDLE" is not found within
a ⇥ "other" ; the string, then perform no
───────────────────────────────────── ; replacement.
f a ⇥ "other"


f ⇥ Text/replace "NEEDLE" replacement ; If the "NEEDLE" is found within
a ⇥ "beforeNEEDLEafter" ; the string, then replace.
"before${replacement}" ++ Text/replace "NEEDLE" replacement "after" → e
───────────────────────────────────────────────────────────────────────
f a ⇥ e


f ⇥ Text/replace "NEEDLE" replacement ; The string has interpolation, but
a ⇥ "other${x}remainder…" ; the prefix does not match.
"other${x}" ++ Text/replace "NEEDLE" "remainder…" → e
─────────────────────────────────────────────────────
f a ⇥ e


f ⇥ Text/replace "NEEDLE" replacement ; The string has interpolation, and
a ⇥ "beforeNEEDLEafter${x}remainder…" ; the prefix matches.
"before${replacement}" ++ Text/replace "NEEDLE" "after${x}remainder…" → e
─────────────────────────────────────────────────────────────────────────
f a ⇥ e


All of the built-in functions on `Text` are in normal form:


─────────────────────
Text/show ⇥ Text/show


───────────────────────────
Text/replace ⇥ Text/replace


## `List`

The `List` type-level function is in normal form:
Expand Down
8 changes: 8 additions & 0 deletions standard/binary.md
Original file line number Diff line number Diff line change
Expand Up @@ -267,6 +267,10 @@ matching their identifier.
encode(Text/show) = "Text/show"


─────────────────────────────────────
encode(Text/replace) = "Text/replace"


─────────────────────────
encode(Bool) = "Bool"

Expand Down Expand Up @@ -987,6 +991,10 @@ a built-in identifier if it matches any of the following strings:
decode("List/reverse") = List/reverse


─────────────────────────────────────
decode("Text/replace") = Text/replace


───────────────────────────────
decode("Text/show") = Text/show

Expand Down
2 changes: 2 additions & 0 deletions standard/dhall.abnf
Original file line number Diff line number Diff line change
Expand Up @@ -412,6 +412,7 @@ builtin =
/ List-indexed
/ List-reverse
/ Text-show
/ Text-replace
/ Bool
/ True
/ False
Expand Down Expand Up @@ -464,6 +465,7 @@ List-last = %x4c.69.73.74.2f.6c.61.73.74
List-indexed = %x4c.69.73.74.2f.69.6e.64.65.78.65.64
List-reverse = %x4c.69.73.74.2f.72.65.76.65.72.73.65
Text-show = %x54.65.78.74.2f.73.68.6f.77
Text-replace = %x54.65.78.74.2f.72.65.70.6c.61.63.65

; Operators
combine = %x2227 / "/\"
Expand Down
4 changes: 4 additions & 0 deletions standard/shift.md
Original file line number Diff line number Diff line change
Expand Up @@ -489,6 +489,10 @@ The remaining rules are:
↑(d, x, m, Text/show) = Text/show


───────────────────────────────────────
↑(d, x, m, Text/replace) = Text/replace


───────────────────────
↑(d, x, m, Bool) = Bool

Expand Down
4 changes: 4 additions & 0 deletions standard/substitution.md
Original file line number Diff line number Diff line change
Expand Up @@ -470,6 +470,10 @@ The remaining rules are:
Text/show[x@n ≔ e] = Text/show


────────────────────────────────────
Text/replace[x@n ≔ e] = Text/replace


────────────────────
Bool[x@n ≔ e] = Bool

Expand Down
7 changes: 7 additions & 0 deletions standard/type-inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,13 @@ The `Text` show function has the following type:
Γ ⊢ Text/show : Text → Text


The `Text/replace` function has the following type:


───────────────────────────────────────────────────────────────────────────────────────
Γ ⊢ Text/replace : ∀(needle : Text) → ∀(replacement : Text) → ∀(haystack : Text) → Text


The `Text` concatenation operator takes arguments of type `Text` and returns a
result of type `Text`:

Expand Down
4 changes: 4 additions & 0 deletions tests/normalization/success/unit/TextReplaceAbstractA.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{- This test verifies that an implementation correctly permits both the
"replacement" and the "haystack" to be abstract.
-}
λ(x : Text) → λ(y : Text) → Text/replace "a" "-${x}-" "_a_${y}_a_"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand how the haystack can be abstract. Don't we want to replace occurrences of the needle in the abstract section too? i.e. What happens if y is the string "a"?

Copy link
Contributor

@Gabriella439 Gabriella439 Oct 16, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@basile-henry: Oh, I see what you mean. Allowing the haystack to be abstract means that β-reduction is no longer confluent. For right now, just ignore that test and I can put up a change to require the haystack to be non-abstract before further reduction can occur.

Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
λ(x : Text) → λ(y : Text) → "_-${x}-_${y}_-${x}-_"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceEmptyA.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace "" "bar" "foo"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceEmptyB.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"foo"
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace "foo" "bar" "foobazfoo"
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"barbazbar"
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace "👨" "👩" "👨‍👩‍👧‍👦"
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"👩‍👩‍👧‍👦"
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
-- This test verifies that implementations do not perform Unicode normalization
-- before substitution.
Text/replace "a\u{0301}" "b" "\u{00e1}c"
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"ác"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceSimpleA.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace "foo" "bar" "foo"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceSimpleB.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"bar"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceUnicodeA.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace "💣" "💥" "💣💣💣"
1 change: 1 addition & 0 deletions tests/normalization/success/unit/TextReplaceUnicodeB.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"💥💥💥"
1 change: 1 addition & 0 deletions tests/parser/success/builtinsA.dhall
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
, List/indexed
, List/reverse
, Text/show
, Text/replace
, Bool
, True
, False
Expand Down
2 changes: 1 addition & 1 deletion tests/parser/success/builtinsB.dhallb
Original file line number Diff line number Diff line change
@@ -1 +1 @@
$�lNatural/foldmNatural/buildnNatural/isZerolNatural/evenkNatural/oddqNatural/toIntegerlNatural/showpInteger/toDoublelInteger/shownInteger/negatemInteger/clamppNatural/subtractkDouble/showjList/buildiList/foldkList/lengthiList/headiList/lastlList/indexedlList/reverseiText/showdBool��hOptionaldNonegNaturalgIntegerfDoubledTextdListdTypedKinddSort
%�lNatural/foldmNatural/buildnNatural/isZerolNatural/evenkNatural/oddqNatural/toIntegerlNatural/showpInteger/toDoublelInteger/shownInteger/negatemInteger/clamppNatural/subtractkDouble/showjList/buildiList/foldkList/lengthiList/headiList/lastlList/indexedlList/reverseiText/showlText/replacedBool��hOptionaldNonegNaturalgIntegerfDoubledTextdListdTypedKinddSort
2 changes: 1 addition & 1 deletion tests/parser/success/builtinsB.diag
Original file line number Diff line number Diff line change
@@ -1 +1 @@
[4, null, "Natural/fold", "Natural/build", "Natural/isZero", "Natural/even", "Natural/odd", "Natural/toInteger", "Natural/show", "Integer/toDouble", "Integer/show", "Integer/negate", "Integer/clamp", "Natural/subtract", "Double/show", "List/build", "List/fold", "List/length", "List/head", "List/last", "List/indexed", "List/reverse", "Text/show", "Bool", true, false, "Optional", "None", "Natural", "Integer", "Double", "Text", "List", "Type", "Kind", "Sort"]
[4, null, "Natural/fold", "Natural/build", "Natural/isZero", "Natural/even", "Natural/odd", "Natural/toInteger", "Natural/show", "Integer/toDouble", "Integer/show", "Integer/negate", "Integer/clamp", "Natural/subtract", "Double/show", "List/build", "List/fold", "List/length", "List/head", "List/last", "List/indexed", "List/reverse", "Text/show", "Text/replace", "Bool", true, false, "Optional", "None", "Natural", "Integer", "Double", "Text", "List", "Type", "Kind", "Sort"]
1 change: 1 addition & 0 deletions tests/type-inference/success/unit/TextReplaceA.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Text/replace
1 change: 1 addition & 0 deletions tests/type-inference/success/unit/TextReplaceB.dhall
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
∀(needle : Text) → ∀(replacement : Text) → ∀(haystack : Text) → Text