Strings: normalize strings into UTF-8 NFC #150

jkuchar · 2017-10-14T06:03:22Z

bug fix? no
new feature? yes
BC break? yes

implementation of #149

Before merge:

decide what to do with compare/normalize inconsistency

JanTvrdik · 2017-10-14T08:46:07Z

composer.json

@@ -15,7 +15,8 @@
 		}
 	],
 	"require": {
-		"php": ">=7.0"
+		"php": ">=7.0",
+		"ext-intl": "*"


should probably remain only as suggested

What should Strings class do when there is no intl extension?

@hrach could you please elaborate on it more then just thumbs down? I think I know some of your arguments, but want to be sure what is on your mind.

Well, I consider the Strings class as the main functionality of the nette/utils package and therefore it's dependencies should be required. Nothing more, nothing less.

@hrach That's a valid opinion, but it's outside of scope of this PR. If you want to move currently suggested stuff to require you need another PR and do it consistently for all stuff in suggest.

@hrach Agree on that. Strings is the main reason why I use nette\utils.

@JanTvrdik That makes sense and exactly that was my next question. Will do another MR with moving all necessary dependencies into requires... There will be probably more discussion needed as we will need list of core functionality.

fixed in bb3a69c

opened #151

JanTvrdik · 2017-10-14T08:47:59Z

src/Utils/Strings.php

@@ -99,10 +99,13 @@ public static function substring(string $s, int $start, int $length = null): str


 	/**
-	 * Removes special controls characters and normalizes line endings and spaces in UTF-8 string.
+	 * Removes special controls characters and normalizes line endings and converts to NFC form and spaces in UTF-8 string.


The sentence now makes no sense. How about sth like

Removes special controls characters and normalizes line endings, spaces and normal form to NFC in UTF-8 string

👍 has been writing it in a car

fixed in ec16e40

JanTvrdik · 2017-10-14T08:53:22Z

src/Utils/Strings.php

 	 */
 	public static function normalize(string $s): string
 	{
+		// normalize string into utf8 NFC form


„UTF-8“ instead of „utf8“, or maybe even better to omit altogether

the word „string“ is unnecessary

How about sth like „convert to NFC normal form “?

fixed in b6dd526

jkuchar · 2017-10-14T09:16:15Z

@JanTvrdik Thanks for comments 😃

… user do not need to use Strings class at all)

jkuchar · 2017-10-16T09:17:39Z

@JanTvrdik @hrach thanks for nice argument-based discussion!

I have pushed fixes into this pull request, are there any more comments on this?

jkuchar · 2017-10-16T09:21:22Z

src/Utils/Strings.php

@@ -265,6 +268,9 @@ public static function capitalize(string $s): string
 	 */
 	public static function compare(string $left, string $right, int $len = null): bool
 	{
+		$left = \Normalizer::normalize($left, \Normalizer::FORM_D); // form NFD is faster
+		$right = \Normalizer::normalize($right, \Normalizer::FORM_D); // form NFD is faster


Does it make sense to call Normalizer directly? Or would we prefer to call self::normalize(). This will also normalizer line ending and some other stuff. I'm not sure on which level should be normalization done here in compare.

Current behaviour (proposed by @dg) causes inconsistencies as it is possible to get into situation where:

Strings::compare($a, $b); // FALSE Strings::normalize($a) === Strings::normalize($b); // TRUE

Is this expected behaviour? It seems as confusing behaviour to me.

Is it possible to call it after substring() ?

No. Substring operates on code units, normalize may change length in code units (imho).

jkuchar · 2017-10-22T18:18:59Z

The last thing to sort out is the following inconsistency. Current behaviour (proposed by @dg) causes inconsistencies as it is possible to get into situation where:

Strings::compare($a, $b); // FALSE
Strings::normalize($a) === Strings::normalize($b); // TRUE

It seems as confusing behaviour to me. What are your opinions on this?

JanTvrdik · 2017-10-23T08:30:15Z

Current behaviour (proposed by @dg) causes inconsistencies

This seems OK to me. It would be quite confusing if Strings::compare("\r\n", "\t \x00\n") returned true. The compare function is just case-insensitive variant of ===.

jkuchar · 2017-10-23T19:22:54Z

👍 Ok, it makes sense. So, now it looks like ready to merge,

dg · 2017-10-24T00:39:58Z

Thank you

jkuchar force-pushed the 149-normalize-utf8-strings branch from 05be0ad to 39a924d Compare October 14, 2017 06:04

Strings: normalizes strings into UTF-8 NFC (closes nette#149)

1231f4c

jkuchar force-pushed the 149-normalize-utf8-strings branch from 39a924d to 1231f4c Compare October 14, 2017 06:09

JanTvrdik reviewed Oct 14, 2017

View reviewed changes

JanTvrdik and others added 2 commits October 14, 2017 21:58

fixup! typo

ec16e40

fixup! typo

b6dd526

jkuchar force-pushed the 149-normalize-utf8-strings branch from f6d2295 to b6dd526 Compare October 14, 2017 20:01

fixup! composer: moved intl extension into suggested dependencies (as…

bb3a69c

… user do not need to use Strings class at all)

jkuchar force-pushed the 149-normalize-utf8-strings branch from 11c1905 to bb3a69c Compare October 16, 2017 09:14

jkuchar commented Oct 16, 2017

View reviewed changes

jkuchar mentioned this pull request Oct 16, 2017

Composer: Which dependencies should be required? #151

Closed

3 tasks

jkuchar changed the title ~~Strings: normalizes strings into UTF-8 NFC~~ Strings: normalize strings into UTF-8 NFC Oct 22, 2017

dg merged commit 6b858f6 into nette:master Oct 24, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strings: normalize strings into UTF-8 NFC #150

Strings: normalize strings into UTF-8 NFC #150

jkuchar commented Oct 14, 2017 •

edited

Loading

JanTvrdik Oct 14, 2017

jkuchar Oct 14, 2017

jkuchar Oct 14, 2017

hrach Oct 14, 2017

JanTvrdik Oct 14, 2017

jkuchar Oct 16, 2017 •

edited

Loading

jkuchar Oct 16, 2017

jkuchar Oct 16, 2017 •

edited

Loading

JanTvrdik Oct 14, 2017

jkuchar Oct 14, 2017

jkuchar Oct 14, 2017

JanTvrdik Oct 14, 2017

jkuchar Oct 14, 2017

jkuchar Oct 14, 2017

jkuchar commented Oct 14, 2017

jkuchar commented Oct 16, 2017

jkuchar Oct 16, 2017 •

edited

Loading

dg Oct 16, 2017

JanTvrdik Oct 17, 2017

jkuchar commented Oct 22, 2017

JanTvrdik commented Oct 23, 2017

jkuchar commented Oct 23, 2017

dg commented Oct 24, 2017

Strings: normalize strings into UTF-8 NFC #150

Strings: normalize strings into UTF-8 NFC #150

Conversation

jkuchar commented Oct 14, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkuchar Oct 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkuchar Oct 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkuchar commented Oct 14, 2017

jkuchar commented Oct 16, 2017

jkuchar Oct 16, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jkuchar commented Oct 22, 2017

JanTvrdik commented Oct 23, 2017

jkuchar commented Oct 23, 2017

dg commented Oct 24, 2017

jkuchar commented Oct 14, 2017 •

edited

Loading

jkuchar Oct 16, 2017 •

edited

Loading

jkuchar Oct 16, 2017 •

edited

Loading

jkuchar Oct 16, 2017 •

edited

Loading