Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

[Console]Improve formatter for double-width character #10714

Closed
wants to merge 7 commits into from

4 participants

@denkiryokuhatsuden
Q A
Bug fix? yes
New feature? no
BC breaks? yes for one that expecting current broken output
Deprecations? no
Fixed tickets
Tests pass? yes
License MIT
Doc PR

EDIT: fixed the table above not to remove irrelevant line

As mb_strlen just returns "how many chars in the string",
formatting with double-width character is bit broken.

The test I add is skipped when mbstring extension is not loaded.
I'm afraid if some of you cannot properly display japanese string.
(表示するテキスト just means "Some text to display")

@denkiryokuhatsuden denkiryokuhatsuden [Console]Add failing test for double-width char
`$formatter->formatBlock()` formats a broken block
when some double-width character is set.
b756df8
@denkiryokuhatsuden denkiryokuhatsuden [Console]use mb_strwidth instead of mb_strlen
Some character has doubled-width so `mb_strlen` does not return actual string width (just returns count of chars).
By using `mb_strwidth` (http://www.php.net/manual/en/function.mb-strwidth.php),
it can property measure the actual width of string.
1899590
@denkiryokuhatsuden denkiryokuhatsuden [Console]Add similar test for Application#renderException
When `Application` renderes Exception, it also writes broken block when
message contains double-width (more generally, multi-byted) character.

This commit fixes above.
15e136d
@denkiryokuhatsuden denkiryokuhatsuden Fixed CS (thanks to fabbot)
dece79f
@denkiryokuhatsuden

I found that rendering error uses logic independent of FormatterHelper, and it has similar problem.
15e136d is commit to fix them.

src/Symfony/Component/Console/Application.php
@@ -722,7 +722,37 @@ public function renderException($e, $output)
return strlen($string);
}
- return mb_strlen($string, $encoding);
+ return mb_strwidth($string, $encoding);
+ };
+
+ // str_split is not suitable for multi-byte characters, we should use preg_split to get char array properly.
+ // additionally, array_slice() is not enough as some character has doubled width.
+ // we need a function to split string not by character count but by string width
+ $str_split_width = function ($string, $width) {
@stof Collaborator
stof added a note

please make it a private method rather than a closure which need to be recreated by PHP for each call to renderException

above $str_split_width, $strlen also exists as closure. Shall I move $strlen as private method either?

and could you please give advice for method naming,

  • $strlen
    -> private strlen for consistency (Helper has a function with same name)
    -> private strwidth as this does not return length of string but width

  • $str_split_width
    -> how about private splitStringByWidth()?

@stof Collaborator
stof added a note

agreed about splitStringByWidth rather than $str_split_width.

and for $strlen, it could indeed be renamed to a private method stringWidth as it does not compute the length anymore after your change

Thanks for advice.
Shall I leave the name of Helper#strlen unchanged (which I changed its behavior, too) unlike Application's, as changing that name introduces BC break?

@stof Collaborator
stof added a note

yeah, it cannot be renamed as it is protected

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Symfony/Component/Console/Application.php
@@ -722,7 +722,37 @@ public function renderException($e, $output)
return strlen($string);
}
- return mb_strlen($string, $encoding);
+ return mb_strwidth($string, $encoding);
+ };
+
+ // str_split is not suitable for multi-byte characters, we should use preg_split to get char array properly.
+ // additionally, array_slice() is not enough as some character has doubled width.
+ // we need a function to split string not by character count but by string width
+ $str_split_width = function ($string, $width) {
+ if (!function_exists('mb_strwidth')) {
+ return str_split($string, $width);
+ }
+ $originalEncoding = mb_detect_encoding($string);
@stof Collaborator
stof added a note

you need to handle the case of returning false (i.e. not able to detect the encoding)

I changed the code not to do anything about converting encode if mb_detect_encoding returns false.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
src/Symfony/Component/Console/Application.php
((13 lines not shown))
+ }
+
+ return mb_strwidth($string, $encoding);
+ }
+
+ private function splitStringByWidth($string, $width)
+ {
+ // str_split is not suitable for multi-byte characters, we should use preg_split to get char array properly.
+ // additionally, array_slice() is not enough as some character has doubled width.
+ // we need a function to split string not by character count but by string width
+ if (!function_exists('mb_strwidth')) {
+ return str_split($string, $width);
+ }
+ $originalEncoding = mb_detect_encoding($string);
+
+ $convertedString = false === $originalEncoding ? $string : mb_convert_encoding($string, 'utf8', $originalEncoding);
@stof Collaborator
stof added a note

To be consistent with stringWidth, you should fallback to str_split if mb_string does not support the encoding

Thanks, I've overlooked the code in stringWidth. I've changed variable's name, too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@fabpot
Owner

Thank you @denkiryokuhatsuden.

@fabpot fabpot referenced this pull request from a commit
@fabpot fabpot bug #10714 [Console]Improve formatter for double-width character (den…
…kiryokuhatsuden)

This PR was squashed before being merged into the 2.3 branch (closes #10714).

Discussion
----------

[Console]Improve formatter for double-width character

| Q             | A
| ------------- | ---
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | yes for one that expecting current broken output
| Deprecations? | no
| Fixed tickets |
| Tests pass?   | yes
| License       | MIT
| Doc PR        |

EDIT: fixed the table above not to remove irrelevant line

As mb_strlen just returns "how many chars in the string",
formatting with double-width character is bit broken.

The test I add is skipped when mbstring extension is not loaded.
I'm afraid if some of you cannot properly display japanese string.
(表示するテキスト just means "Some text to display")

Commits
-------

a52f41d [Console]Improve formatter for double-width character
a29a60d
@fabpot fabpot closed this
@jakzal
Collaborator

@denkiryokuhatsuden on which PHP version have you been testing this?

@denkiryokuhatsuden

@jakzal tested on php5.5. Are there some tests failing?

@jakzal
Collaborator

@denkiryokuhatsuden yes, they were never passing on PHP 5.5. See #10897.

@fabpot fabpot referenced this pull request from a commit
@fabpot fabpot bug #10897 [Console] Fix a console test (jakzal)
This PR was merged into the 2.3 branch.

Discussion
----------

[Console] Fix a console test

| Q             | A
| ------------- | ---
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | -
| License       | MIT
| Doc PR        | -

re #10714

Every wide character in the fixture file is actually 3 ansi-characters long.

Commits
-------

61108b9 Disable 5.6 until it is stable again.
8cadb49 Update the fixtures.
7b93db5
@fabpot fabpot referenced this pull request from a commit
@fabpot fabpot bug #10899 Explicitly define the encoding. (jakzal)
This PR was merged into the 2.3 branch.

Discussion
----------

Explicitly define the encoding.

| Q             | A
| ------------- | ---
| Bug fix?      | yes
| New feature?  | no
| BC breaks?    | no
| Deprecations? | no
| Tests pass?   | yes
| Fixed tickets | -
| License       | MIT
| Doc PR        | -

Credits for discovering it go to @nicolas-grekas. Cheers!

Travis for PHP 5.6 cannot be enabled yet as there's one more test failing.

re #10714 #10714

Commits
-------

619ff58 Explicitly define the encoding.
6207389
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Apr 15, 2014
  1. @denkiryokuhatsuden

    [Console]Add failing test for double-width char

    denkiryokuhatsuden authored
    `$formatter->formatBlock()` formats a broken block
    when some double-width character is set.
  2. @denkiryokuhatsuden

    [Console]use mb_strwidth instead of mb_strlen

    denkiryokuhatsuden authored
    Some character has doubled-width so `mb_strlen` does not return actual string width (just returns count of chars).
    By using `mb_strwidth` (http://www.php.net/manual/en/function.mb-strwidth.php),
    it can property measure the actual width of string.
Commits on Apr 17, 2014
  1. @denkiryokuhatsuden

    [Console]Add similar test for Application#renderException

    denkiryokuhatsuden authored
    When `Application` renderes Exception, it also writes broken block when
    message contains double-width (more generally, multi-byted) character.
    
    This commit fixes above.
  2. @denkiryokuhatsuden

    Fixed CS (thanks to fabbot)

    denkiryokuhatsuden authored
  3. @denkiryokuhatsuden
  4. @denkiryokuhatsuden
  5. @denkiryokuhatsuden
This page is out of date. Refresh to see the latest.
View
91 src/Symfony/Component/Console/Application.php
@@ -99,7 +99,7 @@ public function setDispatcher(EventDispatcherInterface $dispatcher)
* @param InputInterface $input An Input instance
* @param OutputInterface $output An Output instance
*
- * @return integer 0 if everything went fine, or an error code
+ * @return int 0 if everything went fine, or an error code
*
* @throws \Exception When doRun returns Exception
*
@@ -159,7 +159,7 @@ public function run(InputInterface $input = null, OutputInterface $output = null
* @param InputInterface $input An Input instance
* @param OutputInterface $output An Output instance
*
- * @return integer 0 if everything went fine, or an error code
+ * @return int 0 if everything went fine, or an error code
*/
public function doRun(InputInterface $input, OutputInterface $output)
{
@@ -270,7 +270,7 @@ public function getHelp()
/**
* Sets whether to catch exceptions or not during commands execution.
*
- * @param bool $boolean Whether to catch exceptions or not during commands execution
+ * @param bool $boolean Whether to catch exceptions or not during commands execution
*
* @api
*/
@@ -282,7 +282,7 @@ public function setCatchExceptions($boolean)
/**
* Sets whether to automatically exit after a command execution or not.
*
- * @param bool $boolean Whether to automatically exit after a command execution or not
+ * @param bool $boolean Whether to automatically exit after a command execution or not
*
* @api
*/
@@ -449,7 +449,7 @@ public function get($name)
*
* @param string $name The command name or alias
*
- * @return Boolean true if the command exists, false otherwise
+ * @return bool true if the command exists, false otherwise
*
* @api
*/
@@ -674,8 +674,8 @@ public static function getAbbreviations($names)
/**
* Returns a text representation of the Application.
*
- * @param string $namespace An optional namespace name
- * @param bool $raw Whether to return raw command list
+ * @param string $namespace An optional namespace name
+ * @param bool $raw Whether to return raw command list
*
* @return string A string representing the Application
*
@@ -691,8 +691,8 @@ public function asText($namespace = null, $raw = false)
/**
* Returns an XML representation of the Application.
*
- * @param string $namespace An optional namespace name
- * @param bool $asDom Whether to return a DOM or an XML string
+ * @param string $namespace An optional namespace name
+ * @param bool $asDom Whether to return a DOM or an XML string
*
* @return string|\DOMDocument An XML string representing the Application
*
@@ -708,34 +708,22 @@ public function asXml($namespace = null, $asDom = false)
/**
* Renders a caught exception.
*
- * @param \Exception $e An exception instance
+ * @param \Exception $e An exception instance
* @param OutputInterface $output An OutputInterface instance
*/
public function renderException($e, $output)
{
- $strlen = function ($string) {
- if (!function_exists('mb_strlen')) {
- return strlen($string);
- }
-
- if (false === $encoding = mb_detect_encoding($string)) {
- return strlen($string);
- }
-
- return mb_strlen($string, $encoding);
- };
-
do {
$title = sprintf(' [%s] ', get_class($e));
- $len = $strlen($title);
+ $len = $this->stringWidth($title);
// HHVM only accepts 32 bits integer in str_split, even when PHP_INT_MAX is a 64 bit integer: https://github.com/facebook/hhvm/issues/1327
$width = $this->getTerminalWidth() ? $this->getTerminalWidth() - 1 : (defined('HHVM_VERSION') ? 1 << 31 : PHP_INT_MAX);
$formatter = $output->getFormatter();
$lines = array();
foreach (preg_split('/\r?\n/', $e->getMessage()) as $line) {
- foreach (str_split($line, $width - 4) as $line) {
+ foreach ($this->splitStringByWidth($line, $width - 4) as $line) {
// pre-format lines to get the right string length
- $lineLength = $strlen(preg_replace('/\[[^m]*m/', '', $formatter->format($line))) + 4;
+ $lineLength = $this->stringWidth(preg_replace('/\[[^m]*m/', '', $formatter->format($line))) + 4;
$lines[] = array($line, $lineLength);
$len = max($lineLength, $len);
@@ -744,7 +732,7 @@ public function renderException($e, $output)
$messages = array('', '');
$messages[] = $emptyLine = $formatter->format(sprintf('<error>%s</error>', str_repeat(' ', $len)));
- $messages[] = $formatter->format(sprintf('<error>%s%s</error>', $title, str_repeat(' ', max(0, $len - $strlen($title)))));
+ $messages[] = $formatter->format(sprintf('<error>%s%s</error>', $title, str_repeat(' ', max(0, $len - $this->stringWidth($title)))));
foreach ($lines as $line) {
$messages[] = $formatter->format(sprintf('<error> %s %s</error>', $line[0], str_repeat(' ', $len - $line[1])));
}
@@ -890,7 +878,7 @@ protected function configureIO(InputInterface $input, OutputInterface $output)
* @param InputInterface $input An Input instance
* @param OutputInterface $output An Output instance
*
- * @return integer 0 if everything went fine, or an error code
+ * @return int 0 if everything went fine, or an error code
*/
protected function doRunCommand(Command $command, InputInterface $input, OutputInterface $output)
{
@@ -1125,4 +1113,53 @@ private function findAlternatives($name, $collection, $abbrevs, $callback = null
return array_keys($alternatives);
}
+
+ private function stringWidth($string)
+ {
+ if (!function_exists('mb_strwidth')) {
+ return strlen($string);
+ }
+
+ if (false === $encoding = mb_detect_encoding($string)) {
+ return strlen($string);
+ }
+
+ return mb_strwidth($string, $encoding);
+ }
+
+ private function splitStringByWidth($string, $width)
+ {
+ // str_split is not suitable for multi-byte characters, we should use preg_split to get char array properly.
+ // additionally, array_slice() is not enough as some character has doubled width.
+ // we need a function to split string not by character count but by string width
+
+ if (!function_exists('mb_strwidth')) {
+ return str_split($string, $width);
+ }
+
+ if (false === $encoding = mb_detect_encoding($string)) {
+ return str_split($string, $width);
+ }
+
+ $utf8String = mb_convert_encoding($string, 'utf8', $encoding);
+ $lines = array();
+ $line = '';
+ foreach (preg_split('//u', $utf8String) as $char) {
+ // test if $char could be appended to current line
+ if (mb_strwidth($line.$char) <= $width) {
+ $line .= $char;
+ continue;
+ }
+ // if not, push current line to array and make new line
+ $lines[] = str_pad($line, $width);
+ $line = $char;
+ }
+ if (strlen($line)) {
+ $lines[] = count($lines) ? str_pad($line, $width) : $line;
+ }
+
+ mb_convert_variables($encoding, 'utf8', $lines);
+
+ return $lines;
+ }
}
View
6 src/Symfony/Component/Console/Helper/Helper.php
@@ -45,11 +45,11 @@ public function getHelperSet()
*
* @param string $string The string to check its length
*
- * @return integer The length of the string
+ * @return int The length of the string
*/
protected function strlen($string)
{
- if (!function_exists('mb_strlen')) {
+ if (!function_exists('mb_strwidth')) {
return strlen($string);
}
@@ -57,6 +57,6 @@ protected function strlen($string)
return strlen($string);
}
- return mb_strlen($string, $encoding);
+ return mb_strwidth($string, $encoding);
}
}
View
27 src/Symfony/Component/Console/Tests/ApplicationTest.php
@@ -469,6 +469,33 @@ public function testRenderException()
$this->assertStringEqualsFile(self::$fixturesPath.'/application_renderexception4.txt', $tester->getDisplay(true), '->renderException() wraps messages when they are bigger than the terminal');
}
+ public function testRenderExceptionWithDoubleWidthCharacters()
+ {
+ $application = $this->getMock('Symfony\Component\Console\Application', array('getTerminalWidth'));
+ $application->setAutoExit(false);
+ $application->expects($this->any())
+ ->method('getTerminalWidth')
+ ->will($this->returnValue(120));
+ $application->register('foo')->setCode(function () {throw new \Exception('エラーメッセージ');});
+ $tester = new ApplicationTester($application);
+
+ $tester->run(array('command' => 'foo'), array('decorated' => false));
+ $this->assertStringEqualsFile(self::$fixturesPath.'/application_renderexception_doublewidth1.txt', $tester->getDisplay(true), '->renderException() renderes a pretty exceptions with previous exceptions');
+
+ $tester->run(array('command' => 'foo'), array('decorated' => true));
+ $this->assertStringEqualsFile(self::$fixturesPath.'/application_renderexception_doublewidth1decorated.txt', $tester->getDisplay(true), '->renderException() renderes a pretty exceptions with previous exceptions');
+
+ $application = $this->getMock('Symfony\Component\Console\Application', array('getTerminalWidth'));
+ $application->setAutoExit(false);
+ $application->expects($this->any())
+ ->method('getTerminalWidth')
+ ->will($this->returnValue(32));
+ $application->register('foo')->setCode(function () {throw new \Exception('コマンドの実行中にエラーが発生しました。');});
+ $tester = new ApplicationTester($application);
+ $tester->run(array('command' => 'foo'), array('decorated' => false));
+ $this->assertStringEqualsFile(self::$fixturesPath.'/application_renderexception_doublewidth2.txt', $tester->getDisplay(true), '->renderException() wraps messages when they are bigger than the terminal');
+ }
+
public function testRun()
{
$application = new Application();
View
11 src/Symfony/Component/Console/Tests/Fixtures/application_renderexception_doublewidth1.txt
@@ -0,0 +1,11 @@
+
+
+
+ [Exception]
+ エラーメッセージ
+
+
+
+foo
+
+
View
11 ...ny/Component/Console/Tests/Fixtures/application_renderexception_doublewidth1decorated.txt
@@ -0,0 +1,11 @@
+
+
+ 
+ [Exception] 
+ エラーメッセージ 
+ 
+
+
+foo
+
+
View
12 src/Symfony/Component/Console/Tests/Fixtures/application_renderexception_doublewidth2.txt
@@ -0,0 +1,12 @@
+
+
+
+ [Exception]
+ コマンドの実行中にエラーが
+ 発生しました。
+
+
+
+foo
+
+
View
15 src/Symfony/Component/Console/Tests/Helper/FormatterHelperTest.php
@@ -69,6 +69,21 @@ public function testFormatBlockWithDiacriticLetters()
);
}
+ public function testFormatBlockWithDoubleWidthDiacriticLetters()
+ {
+ if (!extension_loaded('mbstring')) {
+ $this->markTestSkipped('This test requires mbstring to work.');
+ }
+ $formatter = new FormatterHelper();
+ $this->assertEquals(
+ '<error> </error>'."\n" .
+ '<error> 表示するテキスト </error>'."\n" .
+ '<error> </error>',
+ $formatter->formatBlock('表示するテキスト', 'error', true),
+ '::formatBlock() formats a message in a block'
+ );
+ }
+
public function testFormatBlockLGEscaping()
{
$formatter = new FormatterHelper();
Something went wrong with that request. Please try again.