Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

i18n handler checks for existence of wrong mo file #3128

Closed
ddb4github opened this issue Dec 3, 2019 · 7 comments
Closed

i18n handler checks for existence of wrong mo file #3128

ddb4github opened this issue Dec 3, 2019 · 7 comments
Labels
bug Undesired behaviour resolved A fixed issue
Milestone

Comments

@ddb4github
Copy link
Contributor

ddb4github commented Dec 3, 2019

Describe the bug

  • i18n handler(default:vendor/gettext) use 100% cpu and huge memory for non-en locale.
  • include/global_languages.php:177 ignore CACTI_LANGUAGE_HANDLER_NONE setting, and always use 'CACTI_LANGUAGE_HANDLER_OSCAROTERO'
if (empty($l10n_handler)) {
.....

From Windows 10 FireFox, always call thousands i18n handler function. As stack below:

  • CMDPHP PHP GETTEXT MO Backtrace: (/index.php[25]:include(), /include/auth.php[28]:require_once(), /include/global.php[455]:include_once(), /include/global_languages.php[230]:load_gettext_oscarotero(), /include/global_languages.php[278]:Gettext\Translations::__callStatic(), /include/vendor/gettext/src/Translations.php[153]:call_user_func_array(), Gettext\Translations->__call(), /include/vendor/gettext/src/Translations.php[174]:call_user_func(), Gettext\Extractors\Extractor::fromFile(), /include/vendor/gettext/src/Extractors/Extractor.php[18]:Gettext\Extractors\Mo::fromString())

From MacOS 10.15 FireFox, only call thousands i18n handler function once during login. As stack above:

Sometimes, httpd worker crash with:

  • PHP Fatal error: Allowed memory size of 838860800 bytes exhausted (tried to allocate 524288 bytes) in /var/www/html/cacti/include/vendor/gettext/src/Extractors/Mo.php on line 98

Try to set 'l10n_language_handler'(vendor/phpgettext), run timeout as:

  • PHP Fatal error: Maximum execution time of 60 seconds exceeded in /var/www/html/cacti/include/vendor/phpgettext/streams.php on line 113

To Reproduce
Steps to reproduce the behavior:

  1. Prepare Cacti 1.2.8, and FireFox 68.0+
  2. Change FireFox-->Options-->"Choose your preferred language for displaying pages" to "Hindi" or "Chinese"
  3. Clean browser session/cookie for Cacti web site.
  4. Login
  5. Check background httpd service daemon status, and httpd log

apache2/error_log for httpd worker crash case

  • [Tue Dec 03 15:35:19.248589 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP Fatal error: Out of memory (allocated 516526080) (tried to allocate 2097160 bytes) in /var/www/html/cacti/include/vendor/gettext/src/Extractors/Mo.php on line 98, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.294930 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP Stack trace:, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298892 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 1. {main}() /var/www/html/cacti/index.php:0, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298910 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 2. include() /var/www/html/cacti/index.php:25, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298915 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 3. require_once() /var/www/html/cacti/include/auth.php:28, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298919 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 4. include_once() /var/www/html/cacti/include/global.php:455, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298924 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 5. load_gettext_oscarotero() /var/www/html/cacti/include/global_languages.php:230, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298930 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 6. Gettext\Translations::fromMoFile() /var/www/html/cacti/include/global_languages.php:278, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298940 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 7. Gettext\Translations::__callStatic() /var/www/html/cacti/include/global_languages.php:278, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298946 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 8. call_user_func_array:{/var/www/html/cacti/include/vendor/gettext/src/Translations.php:153}() /var/www/html/cacti/include/vendor/gettext/src/Translations.php:153, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298951 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 9. Gettext\Translations->addFromMoFile() /var/www/html/cacti/include/vendor/gettext/src/Translations.php:153, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298956 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 10. Gettext\Translations->__call() /var/www/html/cacti/include/vendor/gettext/src/Translations.php:153, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298961 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 11. call_user_func:{/var/www/html/cacti/include/vendor/gettext/src/Translations.php:174}() /var/www/html/cacti/include/vendor/gettext/src/Translations.php:174, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298966 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 12. Gettext\Extractors\Extractor::fromFile() /var/www/html/cacti/include/vendor/gettext/src/Translations.php:174, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298971 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 13. Gettext\Extractors\Mo::fromString() /var/www/html/cacti/include/vendor/gettext/src/Extractors/Extractor.php:18, referer: http://localhost/cacti/settings.php?tab=general&
    [Tue Dec 03 15:35:19.298975 2019] [php7:notice] [pid 7696] [client 192.168.1.100:56432] PHP 14. explode() /var/www/html/cacti/include/vendor/gettext/src/Extractors/Mo.php:98, referer: http://localhost/cacti/settings.php?tab=general&

apache2/error_log for httpd worker timeout case

  • [Tue Dec 03 20:50:58.808216 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP Fatal error: Maximum execution time of 60 seconds exceeded in /var/www/html/cacti/include/vendor/phpgettext/streams.php on line 113, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808540 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP Stack trace:, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808559 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 1. {main}() /var/www/html/cacti/index.php:0, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808564 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 2. include() /var/www/html/cacti/index.php:25, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808568 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 3. require_once() /var/www/html/cacti/include/auth.php:28, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808575 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 4. include_once() /var/www/html/cacti/include/global.php:461, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808579 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 5. __() /var/www/html/cacti/include/global_arrays.php:128, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808582 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 6. __gettext() /var/www/html/cacti/include/global_languages.php:440, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808590 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 7. gettext_reader->translate() /var/www/html/cacti/include/global_languages.php:396, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808594 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 8. gettext_reader->load_tables() /var/www/html/cacti/include/vendor/phpgettext/gettext.php:254, referer: http://localhost/cacti/index.php
    [Tue Dec 03 20:50:58.808617 2019] [php7:notice] [pid 7463] [client 192.168.1.100:59914] PHP 9. FileReader->read() /var/www/html/cacti/include/vendor/phpgettext/gettext.php:161, referer: http://localhost/cacti/index.php

Desktop:

  • OS: Windows 10, Mac OS 101.5
  • Browser: FireFox 68.2

Web Server:

  • OS: CentOS 7.x, Ubuntu 18.04
@ddb4github ddb4github changed the title Performance: i18n handler use 100% cpu and huge memory for non-ENG locale Performance: i18n handler use 100% cpu and huge memory for non-en locale Dec 3, 2019
@cigamit
Copy link
Member

cigamit commented Dec 4, 2019

You are going to have to provide more detail. First are the a number of requests coming into the server, or are the responses hanging?

What is the value of the Languages Settings on Console > Cofiguration > Settings > General. Have you saved a user preferred language?

select * from settings_user where name='user_language' and user_id = ?

@ddb4github
Copy link
Contributor Author

ddb4github commented Dec 4, 2019

You are going to have to provide more detail. First are the a number of requests coming into the server, or are the responses hanging?

Only one user, I login@Mac10.15, then login@Win10. Next test Win10 at first. For each testing, I clean local cookie and disable cache. Both Win10 and Mac, the first login response time is about 60s

BTW, I enabled all Cacti official plugins for testing.

What is the value of the Languages Settings on Console > Cofiguration > Settings > General. Have you saved a user preferred language?

select * from settings_user where name='user_language' and user_id = ?
settings:
+-----------------------+-------+
| name                  | value |
+-----------------------+-------+
| i18n_default_language | en-US |
| install_language      | en-US |
+-----------------------+-------+
settings_user:
+---------+---------------+-------+
| user_id | name          | value |
+---------+---------------+-------+
|       1 | user_language | en-US |
|       4 | user_language | en-US |
+---------+---------------+-------+

All settings is en-US, but Browser preferred Chinese.

@netniV
Copy link
Member

netniV commented Dec 4, 2019

If a language is set, that should be used over the browser. If it is not using the table setting, this suggests someone changed this functionality.

Also, you can switch between gettext emulators using the new config option.

@ddb4github
Copy link
Contributor Author

ddb4github commented Dec 4, 2019

A possible fixing:

diff -ruEbwB /usr/src/cacti/include/global_languages.php /var/www/html/cacti/include/global_languages.php
--- /usr/src/cacti/include/global_languages.php  2019-12-02 09:42:06.000000000 +0800
+++ /var/www/html/cacti/include/global_languages.php 2019-12-04 12:34:11.449382348 +0800
@@ -139,7 +139,7 @@
    foreach ($plugins as $plugin) {
        $plugin = $plugin['directory'];

-       if (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo')) {
+       if (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $cacti_locale . '.mo')) {
            $path2catalogue = $config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $cacti_locale . '.mo';
        } elseif (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo')) {
            $path2catalogue = $config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo';

Before fixing: Call Mo.php:98: Gettext\Extractors\Mo::fromString:explode: 9378 times
After fixing: only 541 times

Potential performance issue after fixing:
If some plugin have huge of string in i18n files, login page will slowly show again because it want to load all msg from mo files at first access

BTW, all of 18 plugins is disabled, but their mo files are loaded.
Also, background task poller.php load mo files.

@ddb4github
Copy link
Contributor Author

ddb4github commented Dec 4, 2019

Append full diff for:

  1. A typo cause cacti/locales/LC_MESSAGES/X.mo is loaded 18 times, and fall in Mo.php->fromString exec 64.8k times。 After fixing,gettext load correct mo files with Mo.php->fromString exec 5.9k times.
  2. Change empty() to '' checking to support CACTI_LANGUAGE_HANDLER_NONE
@@ -139,7 +139,7 @@
        foreach ($plugins as $plugin) {
                $plugin = $plugin['directory'];

-               if (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo')) {
+               if (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $cacti_locale . '.mo')) {
                        $path2catalogue = $config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $cacti_locale . '.mo';
                } elseif (file_exists($config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo')) {
                        $path2catalogue = $config['base_path'] . '/plugins/' . $plugin . '/locales/LC_MESSAGES/' . $lang2locale[$cacti_locale]['filename'] . '.mo';
@@ -169,12 +169,12 @@
 $l10n_handler = read_config_option('l10n_language_handler');

 // Is the handler defined in the config but not the db?
-if (empty($l10n_handler) && !empty($config['l10n_language_handler'])) {
+if ($l10n_handler == '' && !empty($config['l10n_language_handler'])) {
        l10n_debug('Handler: not specified in settings');
        $l10n_handler = $config['l10n_language_handler'];
 }

-if (empty($l10n_handler)) {
+if ($l10n_handler == '') {
        l10n_debug('Handler: not specified in config, autodetection is now in progress');
        if (file_exists($config['include_path'] . '/vendor/gettext/src/Translator.php')) {
                $l10n_handler = CACTI_LANGUAGE_HANDLER_OSCAROTERO;

@ddb4github
Copy link
Contributor Author

A interesting test result:

  • Server Env: Cacti 1.2.8 with 18 official plugins disabled
  • Browser: FireFox 68 on MacOS 10.15 and Win10.1909
  • Bothe of Http Request and Response Header include Connection: keep-alive
  • Test Case:
    -- Access index.php;
    -- Login;
    -- Switch tab several times between 'console' and 'logs'

Result for MacOS

  • global_languages.php is executed only once (I added cacti_log to check)
  • httpd processor pid is different for each PHP request(I enable'pid' output for httpd access_log)
  • First index.php(before login) response time is about 7s,
  • Other response time is about 300ms
  • After a while(maybe several hours), global_languages.php is executed for each request.

Result for Win10.1909

  • global_languages.php is executed for each request (I added cacti_log to check)
  • httpd processor pid is different for each PHP request(I enable'pid' output for httpd access_log)
  • All of response time are about 7s,

Finally
I guess that PHP more like MacOS, and judge the continuously request as one PHP process, and include_once` is still efficient. (Maybe some php/apache configuration)
But for Win10.1909, it is not a MFN of PHP

@cigamit cigamit added the bug Undesired behaviour label Dec 6, 2019
cigamit added a commit that referenced this issue Dec 6, 2019
Performance: i18n handler use 100% cpu and huge memory for non-en locale
@cigamit cigamit added the resolved A fixed issue label Dec 6, 2019
@cigamit cigamit added this to the v1.2.8 milestone Dec 6, 2019
@cigamit
Copy link
Member

cigamit commented Dec 6, 2019

Resolved Thanks!

@cigamit cigamit closed this as completed Dec 6, 2019
@netniV netniV changed the title Performance: i18n handler use 100% cpu and huge memory for non-en locale i18n handler checks for existence of wrong mo file Dec 7, 2019
@github-actions github-actions bot locked and limited conversation to collaborators Jun 30, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Undesired behaviour resolved A fixed issue
Projects
None yet
Development

No branches or pull requests

3 participants