enhance Inflector helper with ascii function #464

tonydspaniard · 2013-05-30T16:19:28Z

closes #364

Edit:
Seems to me that transliteration without PECL or the new Transliterator class (PHP >= 5.4), is something that a simple array won't do.

I think is ok on the first round, but if we wish to adopt transliteration the right way, we should consider some good rework on this matter.

References:

https://drupal.org/project/transliteration
http://php.net/manual/es/transliterator.transliterate.php
https://doc.wikimedia.org/mediawiki-core/master/php/html/UtfNormal_8php_source.html

qiangxue · 2013-05-30T16:47:35Z

framework/yii/helpers/base/Inflector.php

+	 * @param array $replace the characters to be replaced by spaces.
+	 * @return string the translated
+	 */
+	public static function ascii($string, $replace = array())


I think we don't need this $replace because it does something that has nothing to do with ascii.

yes, it is true, but when you translate something like: I'll be back, you get Ill be back which will make a wrong slug: Ill-be-back instead of I-ll-be-back if we use replacement tokens

Why I'll be back becomes Ill-be-back? Isn't ' an ASCII char already?

You are right, I get your point, that should be work of the slug function.

…moved dependency on ascii function

* upstream: Fixed build break. Fixes issue yiisoft#194: Added Application::catchAll. added missing default to getValue in boostrap tabs \yii\widgets\Menu improvement Fixes issue yiisoft#467: allow view file to be absent as long as the themed version exists. better auto scrolling. Fixes issue yiisoft#446: automatically scroll to first error.

qiangxue · 2013-06-01T21:44:57Z

framework/yii/helpers/base/Inflector.php

 	 */
-	public static $transliteration = array(
+	protected static $transliteration = array(


Why turning this into protected?

imo they should all be customizable.

Also transliteration prop have incorrect map. Please check this http://iamseanmurphy.com/creating-seo-friendly-urls-in-php-with-url-slug/ .

I have inspired my self on that link, how come is incorrect?

@tonydspaniard For example there is no rule for cyrilic а. Also for example rule '/ъ|ь|Ъ|Ы|Ь/' => '' incorrect because Ы should be Y. If you inspired by this link we should take these rules carefully ;)

@tonydspaniard I think its good idea to make some kind of converter to automatic convert rules from link to this format.

I agree with @cebe, working on this

* upstream/master: (91 commits) fixed init.bat paths renamed backstage → backend Fixed test break. Response WIP coding style fix. [1] Redone missing code. [2] Added empty line at the end of file [3] Removed exception. Updated code style. braces on the same line for control statements. Removed unused columsn from find constraint sql. Fixed typo. Added extra schema check for when a foreign table is not in the same schema. Updated indentation to conform to other classes. Multilevel Items Fixes issue yiisoft#514. Removed the config setting that should not have been commited. Removed false exception catching. Removed custom pgsql PDO and added defaultSchema as public property. CS fixes. Minor refactoring of DbMessageSource. Response WIP Minor refactoring of t(). Added Jsonable support. Fixes yiisoft#13. Implement DB message source. Yii::t() minor fix. ...

cebe · 2013-10-22T23:16:42Z

@tonydspaniard as we are on PHP 5.4 now, can you check if/whats obsolete here now?

samdark · 2013-10-23T08:10:18Z

@cebe transliteration is now available in 5.4 intl: http://www.php.net/manual/en/class.transliterator.php

iJackUA · 2013-10-23T10:45:02Z

@samdark , @cebe it is availabe in Intl, but as I understand it is supposed that Intl could be not installed (as for Messages you are making a fallback functionality) so it seems that fallback for translitaration is also (still) needed.

samdark · 2013-10-23T10:49:08Z

We're creating a fallback for English only where transliteration isn't needed.

iJackUA · 2013-10-23T11:17:16Z

I understand it, so why not to make slug function with both options something like

    public static function slug($string, $replacement = '-')
    {
        if(extension_loaded(intl)) {

            $options = "Any-Latin; NFD; [:Nonspacing Mark:] Remove; NFC; [:Punctuation:] Remove;";
            //best options should be evaluated, just a quick copy-paste            
            $string = transliterator_transliterate($options, $string);
            $string = preg_replace('/[-\s]+/', '-', $string);
            return trim($string, '-');

        } else {
        $map = static::$transliteration + [
                '/[^\w\s]/' => ' ',
                '/\\s+/' => $replacement,
                '/(?<=[a-z])([A-Z])/' => $replacement . '\\1',
                str_replace(':rep', preg_quote($replacement, '/'), '/^[:rep]+|[:rep]+$/') => ''
            ];
        return preg_replace(array_keys($map), array_values($map), $string);
        }
    }

tonydspaniard · 2013-12-20T13:13:44Z

@cebe back on track with this, will review again all what is needed and what is not. @iJackUA fallback proposal seems good to me.

* upstream: (2012 commits) doc fix. Changed the signature of `urlCreator` and button creators for `yii\gridview\ActionColumn` parameter adjustment. The signature for `yii\gridview\ActionColumn::urlCreator` is changed - the `$action` parameter is moved to the first Fixed the signature of Schema::findUniqueIndexes(). reverted yiisoft#1598 and added a test for it Fix wrong array index in unique indexes for MySql Making accesors public Get DB unique indexes information Fixes yiisoft#1610: `Html::activeCheckboxList()` and `Html::activeRadioList()` will submit an empty string if no checkbox/radio is selected Gii should keep horizontal layout Documentation at "yii\authclient" updated. Doc comments at "yii\authclient" updated. Auth clients "Choice" doc comments updated. Auth clients "Choice" widget javascript advanced. Bootstrap's dropdown encodes also trailing caret tag Auth clients "Choice" widget markup updated. Gii should keep horizontal layout extended from codeception testcase, added docs Auth clients for Facebook, GitHub, LinkedIn added. ... Conflicts: framework/yii/helpers/BaseInflector.php tests/unit/framework/helpers/InflectorTest.php

… its place

tonydspaniard · 2013-12-26T20:08:49Z

After reviewing and checking lots of sources out there, I couldn't find a really good and simple method to transliterate text. iconv fails on cyrillic so I thought about Unicode char code replacements... The best solution so far was the one created on Drupal that was based on UtfNormal from media wiki.

I have added the TransliteratorHelper class for you guys to review it. It could be that the data to transliterate is at the wrong location transliteration/data or is too much for the framework. Please, any feedback is greatly appreciated @yiisoft/core-developers

Edit: TransliteratorHelper fallsback into intl extension if installed.

qiangxue · 2013-12-26T20:33:22Z

I just reviewed the code. I think it's a bit too much including transliteration/data for fallback purpose. The situation is a bit similar to the message formatting where we use intl if possible and the fallback just makes things work but not perfectly correct.

It seems to me @iJackUA's approach is good enough.

What do you think?

tonydspaniard · 2013-12-26T20:52:59Z

Yeah... I thought about transliteration/data was too much, but was the best approach so far. I will rollback to previous and use @iJackUA's approach updating $transliteration values to fit the replacements as good as possible according to UTF-8 tables referred before. We can always update those values.

Thanks @qiangxue

iJackUA · 2013-12-27T08:36:35Z

if I understand correctly $transliteration currently will work only with Latin character
It does not contain Cyrillic for example - so it returns nothing (empty string) if I will do something like ...::slufigy('Привет Йии фрэймворк')
and

$options = "Any-Latin; NFD; [:Nonspacing Mark:] Remove; NFC; [:Punctuation:] Remove;";
$string = transliterator_transliterate($options, $string);

makes much more right now - it transliterate any language (at least I have tried with Russian) to Latin characters

in my test with string
Привет Hello Йии Framework ! Как дела ? How it goes ?
I have such output (the first one is expected and considered correct)
INTL : Privet-Hello-Jii-Framework-Kak-dela-How-it-goes
Inflector::slug : Hello-Framework-How-it-goes

btw should there be an option for lowercasing the string in slug ? As in most cases it is used for Seo Urls and to remove inconsistency I prefere to make url all lowercase

samdark · 2013-12-27T10:07:29Z

Lowercasing it is a good idea. transliterator_transliterate is only available when there's intl installed. Since we're requiring intl for any advanced message translation (such as plurals) anyway I think the good solution is to use intl if available and fallback to English-only it not.

tonydspaniard · 2013-12-27T15:31:20Z

@iJackUA I am going to try to include the Cyrillic reference but do not expect great results as with the helper i have to remove. For those interested on using a more robust transliteration fallback, please check (done versions for both Yii1 + Yi2): https://github.com/2amigos/transliteration-helper

@samdark I guess add a bool parameter to the function won't hurt.

* upstream: (21 commits) Fixes yiisoft#1643: Added default value for `Captcha::options` Fixes yiisoft#1654: Fixed the issue that a new message source object is generated for every new message being translated Allow dash char in ActionColumn’s button names. Added SecurityTest. fixed functional test when enablePrettyUrl is false. fixed composer.json minor doc fix. Fixes yiisoft#1634: Use masked CSRF tokens to prevent BREACH exploits Use better random CSRF token. GII unique indexes avoid autoIncrement columns updated debug retry params. Added sleep(). Added unit test for ActiveRecord::updateAttributes(). Fixes yiisoft#1641: Added `BaseActiveRecord::updateAttributes()` Fixed yiisoft#1504: Debug toolbar isn't loaded successfully in some environments when xdebug is enabled Mongo README.md updated. Fixes yiisoft#1611: Added `BaseActiveRecord::markAttributeDirty()` Number validator was missing Fixes yiisoft#1638: prevent table names from being enclosed within curly brackets twice. Unique indexes rules for single columns into array ...

tonydspaniard · 2013-12-27T17:10:15Z

@qiangxue please review new approach. Tested both with intl and no intl extension. Both returned the same results.

Updated char map and fallback transliteration approach.

qiangxue · 2013-12-27T20:21:10Z

framework/yii/helpers/BaseInflector.php

-			];
-		return preg_replace(array_keys($map), array_values($map), $string);
+		// ensure UTF-8 and remove invalid UTF-8 chars.
+		$string = mb_convert_encoding((string) $string, 'UTF-8', mb_list_encodings());


Is this really needed? Or is it correct to do?

I tend to be very defensive always on code. Maybe it is not needed. I can remove it.

* upstream: Fixed CSRF token masking issue. improved error message of calling invalid scope method. Fixed repo URL Fixes yiisoft#1650: Added Connection::pdoClass. code style fix. added changelog codestyle fix improved checkIntegrity method Modified extension guidlines fix sphinx command signature fixed bug with forgotten param, fixed behavior for one table integrity fixed sequence reset added postgresql features to reset seq/check integrity

enhance Inflector helper with ascii function

qiangxue · 2013-12-28T15:28:59Z

Thanks!

enhance Inflector helper with ascii function

ea8b6e7

qiangxue reviewed May 30, 2013
View reviewed changes

Antonio Ramirez added 4 commits June 1, 2013 17:25

updated transliteration (greek + russian + german + ukrainian) and re…

10c52d7

…moved dependency on ascii function

fix functionality, keep each function to do what it supposed to do

e4e1871

fix test

9ff2580

qiangxue reviewed Jun 1, 2013
View reviewed changes

tonydspaniard added 3 commits December 26, 2013 13:51

added TransliteratorHelper class

fc9fb80

removed ascii method and use the new TransliteratorHelper::process at…

26330b9

… its place

tonydspaniard added 4 commits December 27, 2013 18:04

remove TransliteratorHelper

b7904c4

refactored inflector's slug and improved fallback char map

08aaeda

update phpDoc

61591ea

qiangxue reviewed Dec 27, 2013
View reviewed changes

tonydspaniard added 2 commits December 28, 2013 09:27

remove UTF-8 string conversion

8c1715d

qiangxue added a commit that referenced this pull request Dec 28, 2013

Merge pull request #464 from tonydspaniard/364-toAscii

f18e530

enhance Inflector helper with ascii function

qiangxue merged commit f18e530 into yiisoft:master Dec 28, 2013

tonydspaniard deleted the 364-toAscii branch February 5, 2014 13:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

enhance Inflector helper with ascii function #464

enhance Inflector helper with ascii function #464

tonydspaniard commented May 30, 2013

qiangxue May 30, 2013

tonydspaniard May 31, 2013

qiangxue May 31, 2013

tonydspaniard Jun 1, 2013

qiangxue Jun 1, 2013

cebe Jun 1, 2013

creocoder Jun 1, 2013

tonydspaniard Jun 2, 2013

creocoder Jun 2, 2013

creocoder Jun 2, 2013

tonydspaniard Jun 5, 2013

cebe commented Oct 22, 2013

samdark commented Oct 23, 2013

iJackUA commented Oct 23, 2013

samdark commented Oct 23, 2013

iJackUA commented Oct 23, 2013

tonydspaniard commented Dec 20, 2013

tonydspaniard commented Dec 26, 2013

qiangxue commented Dec 26, 2013

tonydspaniard commented Dec 26, 2013

iJackUA commented Dec 27, 2013

samdark commented Dec 27, 2013

tonydspaniard commented Dec 27, 2013

tonydspaniard commented Dec 27, 2013

qiangxue Dec 27, 2013

tonydspaniard Dec 28, 2013

qiangxue commented Dec 28, 2013

enhance Inflector helper with ascii function #464

enhance Inflector helper with ascii function #464

Conversation

tonydspaniard commented May 30, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cebe commented Oct 22, 2013

samdark commented Oct 23, 2013

iJackUA commented Oct 23, 2013

samdark commented Oct 23, 2013

iJackUA commented Oct 23, 2013

tonydspaniard commented Dec 20, 2013

tonydspaniard commented Dec 26, 2013

qiangxue commented Dec 26, 2013

tonydspaniard commented Dec 26, 2013

iJackUA commented Dec 27, 2013

samdark commented Dec 27, 2013

tonydspaniard commented Dec 27, 2013

tonydspaniard commented Dec 27, 2013

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qiangxue commented Dec 28, 2013