Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[String] a new component for object-oriented strings management with an abstract unit system #33553

Open
wants to merge 2 commits into
base: master
from

Conversation

@nicolas-grekas
Copy link
Member

commented Sep 11, 2019

Q A
Branch? master
Bug fix? no
New feature? yes
Deprecations? no
Tickets -
License MIT
Doc PR -

This is a reboot of #22184 (thanks @hhamon for working on it) and a generalization of my previous work on the topic (patchwork/utf8). Unlike existing libraries (including patchwork/utf8), this component provides a unified API for the 3 unit systems of strings: bytes, code points and grapheme clusters.

The unified API is defined by the AbstractString class. It has 2 direct child classes: BinaryString and AbstractUnicodeString, itself extended by Utf8String and GraphemeString.

All objects are immutable and provide clear edge-case semantics, using exceptions and/or (nullable) types!

Two helper functions are provided to create such strings:

new GraphemeString('foo') == u('foo'); // when dealing with Unicode, prefer grapheme units
new BinaryString('foo') == b('foo');

GraphemeString is the most linguistic-friendly variant of them, which means it's the one ppl should use most of the time when dealing with written text.

Future ideas:

  • improve tests
  • add more docblocks (only where they'd add value!)
  • consider adding more methods in the string API (is*()?, *Encode()?, etc.)
  • first class Emoji support
  • merge the Inflector component into this one
  • use width() to improve truncate() and wordwrap()
  • move method slug() to a dedicated locale-aware service class
  • propose your ideas (send PRs after merge)

Out of (current) scope:

  • what intl provides (collations, transliterations, confusables, segmentation, etc)

Here is the unified API I'm proposing in this PR, borrowed from looking at many existing libraries, but also Java, Python, JavaScript and Go.

function __construct(string $string = '');
static function unwrap(array $values): array
static function wrap(array $values): array
function after($needle, bool $includeNeedle = false, int $offset = 0): self;
function afterLast($needle, bool $includeNeedle = false, int $offset = 0): self;
function append(string ...$suffix): self;
function before($needle, bool $includeNeedle = false, int $offset = 0): self;
function beforeLast($needle, bool $includeNeedle = false, int $offset = 0): self;
function camel(): self;
function chunk(int $length = 1): array;
function collapseWhitespace(): self
function endsWith($suffix): bool;
function ensureEnd(string $suffix): self;
function ensureStart(string $prefix): self;
function equalsTo($string): bool;
function folded(): self;
function ignoreCase(): self;
function indexOf($needle, int $offset = 0): ?int;
function indexOfLast($needle, int $offset = 0): ?int;
function isEmpty(): bool;
function join(array $strings): self;
function jsonSerialize(): string;
function length(): int;
function lower(): self;
function match(string $pattern, int $flags = 0, int $offset = 0): array;
function padBoth(int $length, string $padStr = ' '): self;
function padEnd(int $length, string $padStr = ' '): self;
function padStart(int $length, string $padStr = ' '): self;
function prepend(string ...$prefix): self;
function repeat(int $multiplier): self;
function replace(string $from, string $to): self;
function replaceMatches(string $fromPattern, $to): self;
function slice(int $start = 0, int $length = null): self;
function snake(): self;
function splice(string $replacement, int $start = 0, int $length = null): self;
function split(string $delimiter, int $limit = null, int $flags = null): array;
function startsWith($prefix): bool;
function title(bool $allWords = false): self;
function toBinary(string $toEncoding = null): BinaryString;
function toGrapheme(): GraphemeString;
function toUtf8(): Utf8String;
function trim(string $chars = " \t\n\r\0\x0B\x0C\u{A0}\u{FEFF}"): self;
function trimEnd(string $chars = " \t\n\r\0\x0B\x0C\u{A0}\u{FEFF}"): self;
function trimStart(string $chars = " \t\n\r\0\x0B\x0C\u{A0}\u{FEFF}"): self;
function truncate(int $length, string $ellipsis = ''): self;
function upper(): self;
function width(bool $ignoreAnsiDecoration = true): int;
function wordwrap(int $width = 75, string $break = "\n", bool $cut = false): self;
function __clone();
function __toString(): string;

AbstractUnicodeString adds these:

static function fromCodePoints(int ...$codes): self;
function ascii(array $rules = []): self;
function codePoint(int $index = 0): ?int;
function folded(bool $compat = true): parent;
function normalize(int $form = self::NFC): self;
function slug(string $separator = '-'): self;

and BinaryString:

static function fromRandom(int $length = 16): self;
function byteCode(int $index = 0): ?int;
function isUtf8(): bool;
function toUtf8(string $fromEncoding = null): Utf8String;
function toGrapheme(string $fromEncoding = null): GraphemeString;

Case insensitive operations are done with the ignoreCase() method.
e.g. b('abc')->ignoreCase()->indexOf('B') will return 1.

For reference, CLDR transliterations (used in the ascii() method) are defined here:
https://github.com/unicode-org/cldr/tree/master/common/transforms

@nicolas-grekas nicolas-grekas added this to the next milestone Sep 11, 2019

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch 2 times, most recently from 6a1137d to c613662 Sep 11, 2019

@stof

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

no way to unignore case ? and no way to know whether the current object is ignoring case ? This makes the API unusable for code wanting to deal with the string in a case sensitive way while accepting an external string object.

Also should we merge this new component in 4.4, which would mean that its first release is already non-experimental ? We are not allowed to have experimental components in LTS versions, per our LTS policy.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

no way to unignore case ? and no way to know whether the current object is ignoring case ?

->ignoreCase() applies only to the very next call in the fluent API chain. This should answer both your questions. See AbstractString::__clone()

Also should we merge this new component in 4.4

That's something we need to decide indeed. On my side, I think we can make it non-experimental.

@stof

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

and what happens for all methods accepting a string as argument, when passing non-UTF-8 strings to the method on a Utf8String or GraphemeString ?

Regarding the naming, should Utf8String be renamed to highlight it is about code points ? AFAIK, GraphemeString also expects the string to be in UTF-8.

Note that these comments are based purely on your PR description. I haven't looked at the code yet.

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch from c613662 to 8945735 Sep 11, 2019

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

what happens for all methods accepting a string as argument, when passing non-UTF-8 strings to the method on a Utf8String or GraphemeString ?

an InvalidArgumentException is thrown

should Utf8String be renamed to highlight it is about code points ?

I think UTF-8 is more common vocabulary. The previous PR used CodePoint indeed, but this is cryptic to many, and doesn't convey the technical encoding scheme (it could use UTF-16BE/LE, etc., nothing would tell). That's why I think Utf8String is better.

@stof

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

@nicolas-grekas but the whole component is about UTF-8 strings. AFAICT, even BinaryString is not meant to operate on other encodings, as it does not validate that the string is valid UTF-8 before converting it to other implementations.
Btw, this means that naming the component String might also be too generic.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

BinaryString is not meant to operate on other encodings, as it does not validate that the string is valid UTF-8 before converting it to other implementations

It does, dunno why you think otherwise. If you try to convert a random binary string to UTF-8/Grapheme, you'll get an InvalidArgumentException too.

BinaryString is what it tells: it can handle any binary strings and doesn't care about the encoding, like the native PHP string functions, just using an OOP API.

Thus the name of the component.

@fabpot

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

To me, this should go as experimental in 5.0.

@drupol

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2019

Definitely supporting this :-) Nice !

@javiereguiluz

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

Sorry to sound naive, but I can't find in this pull request or the previous one, some brief explanation about when/where should developers use this.

Why/when should we use these classes/methods instead of the normal str_ PHP functions or the mb_str UTF8 functions? Thanks!

Note: I'm not questioning this ... I just want to know where this fits in Symfony developers and Symfony itself. Thanks a lot!

edit: see #33553 (comment)

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch 2 times, most recently from 5ccd5a7 to f9b903b Sep 11, 2019

@javiereguiluz

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

For your consideration, we could turn these 4 methods:

function ensureLeft(string $prefix): self
function ensureRight(string $suffix): self
function padLeft(int $length, string $padStr = ' '): self
function padRight(int $length, string $padStr = ' '): self

Into these 2 methods if we change the order of the arguments:

function padLeft(string $padStr = ' ', int $length = null): self
function padRight(string $padStr = ' ', int $length = null): self

Example:

// BEFORE
$s1 = u('lorem')->ensureLeft('abc');
// $s1 = 'abclorem'

$s2 = u('lorem')->ensureRight('abc');
// $s2 = 'loremabc'

$s3 = u('lorem')->padLeft(8, 'abc');
// $s3 = 'abcabcablorem'

$s4 = u('lorem')->padRight(8, 'abc');
// $s4 = 'loremabcabcab'


// AFTER
$s1 = u('lorem')->padLeft('abc');
// $s1 = 'abclorem'

$s2 = u('lorem')->padRight('abc');
// $s2 = 'loremabc'

$s3 = u('lorem')->padLeft('abc', 8);
// $s3 = 'abcabcablorem'

$s4 = u('lorem')->padRight('abc', 8);
// $s4 = 'loremabcabcab'
@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

when/where should developers use this.

All the time would be fine. e.g. $matches = $string->match('/some-regexp/) is a much more friendly API than preg_match('/some-regexp/', $string, $matches) (even more if you consider error handling).

More specifically, I've observed ppl randomly add an mb_ prefix to string functions and magically expect this to fix their encoding issues. This is way too complex right now, doing it correctly is hard. e.g. the Console component deals with utf-8 strings everywhere, it's not pretty. This component would help a lot there. Twig is another place where strings are heavily manipulated and where graphemes are missing actually. It would benefit from the component too.

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 11, 2019

For your consideration, we could turn these 4 methods:
Into these 2 methods if we change the order of the arguments:

This would be totally unexpected to me. I've seen no other libraries have this API and I'm not sure it works actually.

is in your plans that the methods returning self return a new mutated reference keeping the original one intact? If not/yes, why?

Absolutely! That's critical design concern, not just an implementation detail :) I added a note about it in the desription. Thanks for asking.

@azjezz
Copy link
Contributor

left a comment

This is great, i believe this would make it easier for developers to deal with string encoding, just few notes about method naming :)

src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
@javiereguiluz

This comment has been minimized.

Copy link
Member

commented Sep 11, 2019

@nicolas-grekas thanks for the explanation. It's perfectly clear now!

Another question: some methods are called "left", "right" instead of "prefix/suffix" or "start/end". What happens when the text is Arabic/Persian/Hebrew and uses right-to-text direction? For example, trimRight() removes things at the end of English text ... but at the beginning of Arabic text?

@leofeyer

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2019

We have been using tchwork/utf8 in Contao for years and it really is essential if you work with multiple languages beyond the ASCII character range. So +1 for adding this in Symfony and keep up the good work @nicolas-grekas. 👍

@Devristo

This comment has been minimized.

Copy link
Contributor

commented Sep 11, 2019

It looks amazing. I am curious how it would work together with the rest of the ecosystem. Lets say compatibility with doctrine, intl, symfony/validator, etc? I am sure it will take time before it trickles down to other components, but the future seems bright ;)

@azjezz
Copy link
Contributor

left a comment

i suggest adding AbstractString::contains(string ...$needles): bool, where it returns true in case the string contains one of the needles.

if ($text->contains(...$blacklisted)) {
  echo 'nope!';
}
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/AbstractString.php Outdated Show resolved Hide resolved

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch 6 times, most recently from 3706653 to d7dce2f Sep 18, 2019

@Wirone
Copy link

left a comment

Some additional comments from me.

src/Symfony/Component/String/Utf8String.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/Utf8String.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/Utf8String.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/Utf8String.php Outdated Show resolved Hide resolved
src/Symfony/Component/String/Utf8String.php Outdated Show resolved Hide resolved

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch 5 times, most recently from 58dc2b4 to 65fad9e Sep 18, 2019

@nicolas-grekas nicolas-grekas changed the base branch from 4.4 to master Sep 20, 2019

@nicolas-grekas nicolas-grekas modified the milestones: next, 5.0 Sep 20, 2019

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch from 65fad9e to f1ad676 Sep 20, 2019

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 20, 2019

The PR now targets master and has @experimental annotations. Having PHP 7.2 as the minimum version allows using PREG_UNMATCHED_AS_NULL, that's been my main motivation here :)

PR ready.

@GrahamCampbell

This comment has been minimized.

Copy link
Contributor

commented Sep 20, 2019

image

This should probs be updated then. ;)

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch from f1ad676 to 6266ff6 Sep 20, 2019

@nicolas-grekas nicolas-grekas force-pushed the nicolas-grekas:string-component branch from 6266ff6 to 4e56c49 Sep 20, 2019

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 20, 2019

I figured out we could remove the g() helper and keep only u() and b(), by making u() (for Unicode) return a GraphemeString. PR updated (and green).

@nicolas-grekas

This comment has been minimized.

Copy link
Member Author

commented Sep 20, 2019

  • 👍 BinaryString + Utf8String + GraphemeString
  • ❤️ ByteString + CodePointString + GraphemeString

🏑

@javiereguiluz

This comment has been minimized.

Copy link
Member

commented Sep 20, 2019

Would you consider a third option?

ByteString + Utf8String + GraphemeString

}
}
return new static($string);

This comment has been minimized.

Copy link
@stof

stof Sep 20, 2019

Member

this is actually broken if someone calls AbstractUnicodeString::fromCodePoints as it will try to instantiate an abstract class

This comment has been minimized.

Copy link
@nicolas-grekas

nicolas-grekas Sep 20, 2019

Author Member

Correct, but the code is not broken: the call is. So this will fail on purpose.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.