[2.1] Feature/unidecoder #2105

Thinkscape · 2012-08-03T20:24:59Z

WARNING!!!

When opening FILES CHANGED tab on GitHub it takes a longer while to display an can hang your browser**

Rationale

This PR adds Zend\Text\UniDecoder to zf2. It is independent, has no requirements and is an utility class.

UniDecoder is used to convert utf-8 strings into plain-ASCII equivalents, transliterating accents and special characters into their ASCII equivalents. The component is loosely-based on Python Unidecoder and the transliteration is based on a database in form of conversion tables.

I want UniDecoder to find it's way into ZF 2.0.0 because I need it for Console transliteration in case the console is non-utf8 compatible and the Application tries to output multibyte strings. Without transliteration, the whole output will become unreadable and certain special characters can modify remote session settings, clear the screen or throw cursor around the screen.

Basic usage

The class is a static class, similar to Text\Multibyte and offers a single ::decode() method. Below is an example usage:

<?php
use Zend\Text\UniDecoder\UniDecoder;

$text = 'привет, здравствуйте';
echo UniDecoder::decode($text); // outputs: privet, zdravstvuite

- Text\UniDecoder\UniDecoder is now a "static" class, with static ::decode() method - UniDecoder::decode() now accepts second parameter which is a placeholder for broken or unknown characters. - UniDecoder::decode() now checks if it is provided with a scalar value and will throw an exception otherwise. - UniDecoder now handles broken utf-8 strings and will attempt to repair them. - Add tests for all UniDecoder functionality.

Thinkscape · 2012-08-03T20:26:44Z

@weierophinney @DASPRiD

This has been discussed before. It's an utility component which will be used by Zend\Console.
Please merge this PR before RC3.

travisbot · 2012-08-03T20:38:03Z

This pull request fails (merged 45204a8 into 984d799).

travisbot · 2012-08-03T20:51:59Z

This pull request fails (merged 5b59aac into 984d799).

Thinkscape · 2012-08-03T20:56:45Z

I've noticed that PREG behavior changed between PHP 5.4.4 (I've tested the component against) and PHP 5.4.5 (which is used by travis). This makes one of the tests to fail. I will investigate tomorrow and work around it or modify test to match - transliterating broken utf8 strings is a secondary goal.

Maks3w · 2012-08-03T21:27:15Z

I tagged this as 2.1 because any new feature will be released in that version

marc-mabe · 2012-08-04T06:00:29Z

@Thinkscape It looks similar to my work on Zend\Stdlib\StringUtils : https://github.com/marc-mabe/zf2/blob/string/library/Zend/Stdlib/StringUtils.php

Thinkscape · 2012-08-04T06:48:12Z

@Maks3w no. I've discussed it before with @DASPRiD and @weierophinney. It's not a feature, it's a utility class. Please read description above. I need it for console to work properly and console is a 2.0 feature.

Thinkscape · 2012-08-04T06:49:56Z

@marc-mabe Not really :-) Notice that it does not require or depend on any php extension, instead it shuffles bytes and depends on translit tables.

marc-mabe · 2012-08-04T07:35:34Z

@Thinkscape: Sry, can't look into the code currently but doesn't ext/iconv help to transliterate special characters?
Is Zend\Text the right place for it ? - I integrated Zend\Text\MultiByte into Zend\StdLib\StringUtils to make it better available on other components (min dependencies) - It's not ready and not finished discussed because it was postponed to 2.1. Sure it's not the same functionality but there are some overlaps were to use it.
-> Could we discus about it before putting it into 2.0

Thinkscape · 2012-08-04T07:53:47Z

@marc-mabe

iconv('\\TRANSLIT') does not do what this component does.
Zend\Text is the best place, because Zend\Filter and Zend\Validator are due for a cleanup and refactor, but we did not have enough time before RC1.
I know Multibyte and I'm against moving that into Stdlib. Especially if it was to merged with UniDecoder and possibly slugifier (if we ported that too from @DASPRiD)
Stdlib\ArrayUtils and StringUtils work on arrays and strings. Notice that these are built-in datatypes in PHP. Multibyte strings are an external standard. Because PHP uses string for byte data, that's why multibyte strings in PHP are held in string but similarities end here. There are multiple standards for multibyte strings, tens of rules and thousands of characters across different tables. PHP 6 was rumored to introduce built-in, native multibyte string support, but until that time, it's still an external, complex and expensive standard. It must not be squeezed in Stdlib just because there might be many components using it.
UniDecoder consists of 190 new files. That's the main reason it lives inside it's NS under Zend\Text. This allows for easily excluding it, or including on-demand by people.

marc-mabe · 2012-08-04T09:00:45Z

@Thinkscape: Only one note: strings in PHP are a sequence of bytes. This gets a text if you add a character set. PHP 6 only ads a data-type unicode and thats the same as using string+Unicode but now PHP nows about unicode ;)
-> So each text is an "external standard" and since PHP 6 each text not using unicode is an external standard, too.

Maks3w · 2012-08-06T21:16:11Z

Hi,

We have renamed the folder for tests from Zend to ZendTest.

Can you rebase your PR to catch this change?

Thanks in advance.

- orthographic mistakes fixed - hyphens added

…lly populate the required attribute for the input

…eded.

…for example, forms become an array of elements.

- A few test assets had snuck into the tree since this PR was added; moved those under the ZendTest directory - phpunit.xml.dist needed to point to ZendTest directory - likewise with run-tests.sh - Also, if you have run a composer install, you still need access to TestAsset files under ZendTest -- as such, I've made autoloading of that namespace happen in all situations, and register the Zend namespace only if the composer autoloader is not present.

- Done for internal consistency; we recommend using "onEventName" for methods that are event listeners.

- Wildcard route was making the assumption that getPath() could return an empty string. Added a logic path to reset $path to '' when a single slash value is discovered.

Note: If your tests are failing after this patch probably you need to do "php composer.phat update" to update Composer's autoloader with the new lists of namespaces

- Modified tests that omitted "/" path from generated URIs to use them.

- Noted change to use verify_peer by default

- Moved to a listener, and made to update the attributes array only if it is boolean true

- null values passed to elements marked as not required should validate

- trailing whitespace

- Test for null before testing is_string/strlen

Thinkscape · 2012-08-08T11:34:20Z

@Maks3w I've rebased and moved test case to ZendTest.

I've installed php 5.4.5 but this single preg tests that failed before seems like a heisenbug.

Maks3w · 2012-08-11T07:36:39Z

You have a problem with the git history. Use git reflog to rescue your old state and retry the rebase.

DASPRiD · 2012-08-20T12:07:58Z

What's the status of this PR? @Thinkscape do you want me to look into the preg issue?

Thinkscape · 2012-08-20T12:16:34Z

@DASPRiD Yeah, fire away. Works for me.
I've checked again under 5.4.4 and 5.4.5. Tests pass, so I don't know what was the previous behavior about.

weierophinney · 2012-09-14T21:15:34Z

I really, really want to merge this. However, I've spent about 45 minutes trying to resolve merge commits, and it keeps looking like I'm on the verge of breaking something.

If you want to see this in 2.1, can I ask you to please rebase?

juriansluiman · 2012-09-21T12:31:51Z

I cherry picked the relevant commits (please review that!) of this PR and applied them on top of a new branch. This should make it easier to merge. The new PR is #2399

Thinkscape · 2012-09-21T12:35:02Z

Just make sure to include the commit with my unit tests.
If they all pass, everything's jolly.

On Fri, Sep 21, 2012 at 2:31 PM, Jurian Sluiman notifications@github.comwrote:

I cherry picked the relevant commits (please review that!) of this PR and
applied them on top of a new branch. This should make it easier to merge.
The new PR is #2399 #2399

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/2105#issuecomment-8763061.

DASPRiD · 2012-09-21T13:20:12Z

Closing this PR, as @juriansluiman made a new one.

DASPRiD and others added 3 commits August 2, 2012 17:40

Initial slugifier commit

11008a3

Remove Slugifier, update php docblocks.

54bd9b9

Make CS fixer happy.

5b59aac

sasezaki and others added 14 commits August 8, 2012 13:20

remove old Zend\Service scripts from demos

2921413

arabic translation for captcha

ad57f2c

arabic translation for zend validate

d031696

Orthographic mistakes fixed

a24d528

- orthographic mistakes fixed - hyphens added

form annotation builder - if isRequired is true, this will automatica…

425431f

…lly populate the required attribute for the input

adding test to confirm that required attribute is added to element

dfb18c5

fixing for php 5.3.3

fd65d88

Updated the docblocks and made sure the bridge is initialized when ne…

99cbb06

…eded.

[Tests] Rename Zend subdir to ZendTest for PSR-0 compliant

17e9735

adding check for no required value set on password

70d71e5

need to check that the variables aren't already an array. otherwise, …

277822d

…for example, forms become an array of elements.

[Auth] Allow basic resolver to return AuthResult

97c8ebd

typofix

3017e0e

Added suggestion

78639b7

tr and others added 16 commits August 8, 2012 13:28

fixing desending typo in db select.

af20dd6

changed from "file_exists" to "class_exists" in Zend\Filter\Encrypt

c03e48f

fix PSR2 in Zend\Filter\Encrypt

6e672e6

[zendframework#2098] s/execute/onDispatch/

3f88f56

- Done for internal consistency; we recommend using "onEventName" for methods that are event listeners.

[zendframework#2099] Fixes issue in Wildcard route

1d341d6

- Wildcard route was making the assumption that getPath() could return an empty string. Added a logic path to reset $path to '' when a single slash value is discovered.

[Tests] Use Zend_Autoloader when Composer's autoloader is not present.

3c1bd8d

Note: If your tests are failing after this patch probably you need to do "php composer.phat update" to update Composer's autoloader with the new lists of namespaces

Fixes to tests based on URI changes

fa02c69

- Modified tests that omitted "/" path from generated URIs to use them.

[2110] Added README.md entry

d241d82

- Noted change to use verify_peer by default

[zendframework#2079] Move "required" logic to listener

d2eae07

- Moved to a listener, and made to update the attributes array only if it is boolean true

CS fixes

0734af2

Added unit test for ZF2-440 -> set data with null values

53c90fe

[ZF2-440][zendframework#2095] Resolves case of null values

f674fc6

- null values passed to elements marked as not required should validate

[zendframework#2119] CS fixes

763dde4

- trailing whitespace

[zendframework#2119] micro-opt

5fc7d66

- Test for null before testing is_string/strlen

Rebase to latest master, remove redundant test.

3a4e7e7

Thinkscape closed this Aug 8, 2012

Thinkscape reopened this Aug 8, 2012

weierophinney closed this Sep 14, 2012

weierophinney reopened this Sep 14, 2012

juriansluiman mentioned this pull request Sep 21, 2012

Feature/unidecoder new #2399

Closed

DASPRiD closed this Sep 21, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[2.1] Feature/unidecoder #2105

[2.1] Feature/unidecoder #2105

Thinkscape commented Aug 3, 2012

Thinkscape commented Aug 3, 2012

travisbot commented Aug 3, 2012

travisbot commented Aug 3, 2012

Thinkscape commented Aug 3, 2012

Maks3w commented Aug 3, 2012

marc-mabe commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

marc-mabe commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

marc-mabe commented Aug 4, 2012

Maks3w commented Aug 6, 2012

Thinkscape commented Aug 8, 2012

Maks3w commented Aug 11, 2012

DASPRiD commented Aug 20, 2012

Thinkscape commented Aug 20, 2012

weierophinney commented Sep 14, 2012

juriansluiman commented Sep 21, 2012

Thinkscape commented Sep 21, 2012

DASPRiD commented Sep 21, 2012

[2.1] Feature/unidecoder #2105

[2.1] Feature/unidecoder #2105

Conversation

Thinkscape commented Aug 3, 2012

WARNING!!!

Rationale

Basic usage

Thinkscape commented Aug 3, 2012

travisbot commented Aug 3, 2012

travisbot commented Aug 3, 2012

Thinkscape commented Aug 3, 2012

Maks3w commented Aug 3, 2012

marc-mabe commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

marc-mabe commented Aug 4, 2012

Thinkscape commented Aug 4, 2012

marc-mabe commented Aug 4, 2012

Maks3w commented Aug 6, 2012

Thinkscape commented Aug 8, 2012

Maks3w commented Aug 11, 2012

DASPRiD commented Aug 20, 2012

Thinkscape commented Aug 20, 2012

weierophinney commented Sep 14, 2012

juriansluiman commented Sep 21, 2012

Thinkscape commented Sep 21, 2012

DASPRiD commented Sep 21, 2012