Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bad L parameter #549

Open
Guigue79 opened this issue Oct 12, 2017 · 74 comments
Open

Bad L parameter #549

Guigue79 opened this issue Oct 12, 2017 · 74 comments

Comments

@Guigue79
Copy link

Hi
When users or bots try to set Language variable to fr
I have this error
Thu, 12 Oct 2017 09:46:25 +0200 [ERROR] request="eb2a025b3e3ec" component="DmitryDulepov.Realurl.Encoder.UrlEncoder": Bad "L" parameter ("fr") was detected by realurl......
Logs are very very very large and saturate my disk
I try to ignore this param with config.linkVars = L(0-1) but it does not work

@osahner
Copy link

osahner commented Oct 13, 2017

Same here. Logging seems to be messed up this release. One Line has more than 3mb!

@responseinformationsdesign

RealURL also seems to ignore [SYS][systemLogLevel].

Errors get written in TYPO3-Log even [SYS][systemLogLevel] was set to only log fatal errors.

@hannesbochmann
Copy link
Contributor

@responseinformationsdesign
realurl uses the logging system ($TYPO3_CONF_VARS['LOG']) of TYPO3 not the simple system log. System log level has no effect on the logging system. You would need to remove the default FileWriter for log level WARNING after you copied it to the log configuration for log level ERROR. This way you should only have errors in typo3temp/logs/typo3.log. By the way this is no recommended. Warnings shouldn't be ignored. ;-)

@bavarianbytes
Copy link

Same here

@fazzyx
Copy link

fazzyx commented Oct 30, 2017

We have the same situation after upgrading realurl to 2.3.
TYPO3 7.6

@Batman777
Copy link

Typo3 - 6.31
Same problem with huge log files. Will there be a fix in the near future?

You would need to remove the default FileWriter for log level WARNING after you copied it to the log configuration for log level ERROR.

I did not get that. Could you please try in other words?

@hannesbochmann
Copy link
Contributor

@Batman777
Put that in your AdditionalConfiguration.php:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::ERROR]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter'] = $GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::WARNING]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter'];
unset($GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::WARNING]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter']);

@Batman777
Copy link

Awesome .. thank you. Hope this works! :-)

@Batman777
Copy link

This morning the site was online. This afternoon i got a white page.

Log file gives me the following error:
PHP Catchable fatal error: Argument 1 passed to TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter::__construct() must be of the type array, null given, called in /home/XXX/typo3_src-6.2.31/typo3/sysext/core/Classes/Utility/GeneralUtility.php on line 4471 and defined in /home/XXX/typo3_src-6.2.31/typo3/sysext/core/Classes/Log/Writer/FileWriter.php on line 62

After removing the snippet you send me yesterday the site is online again. But now the log file is growing every minute again :-(

@hannesbochmann
Copy link
Contributor

Sorry, the snippet I send you was for TYPO3 7.6 upwards. I checked again in one of my 6.2 projects. That's what I put into AdditionalConfiguration.php:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::ERROR]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter'] =
    $GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::DEBUG]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter'];
unset($GLOBALS['TYPO3_CONF_VARS']['LOG']['writerConfiguration'][\TYPO3\CMS\Core\Log\LogLevel::DEBUG]['TYPO3\\CMS\\Core\\Log\\Writer\\FileWriter']);

This works since weeks.

@Batman777
Copy link

Thank you very much again :-)

@leslawp
Copy link

leslawp commented Nov 10, 2017

Thank you guys for help, especially Hannes! I went into trouble too with huge, few GB log.

@leslawp
Copy link

leslawp commented Nov 10, 2017

Actually it resolves (in T3 7.6) problem with lot of DEBUG entries, caused by every view, but if there is somewhere in the world strange link to our page with bad L parameter then Google will index the whole site since this bad L parameter will glue to every single link. Then lot of ERROR labeled entries will go to the log file. And voila, you have again few GB in few minutes:
This is apache log that shows how it goes:
66.249.64.207 - - [10/Nov/2017:16:07:03 +0100] "GET /index.php?id=1128&L=11 HTTP/1.1" 200 8479 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
What to do? We need different approach... some kind of redirect on bad L or something like this.

@mediaessenz
Copy link

mediaessenz commented Nov 11, 2017

As an other workaround for TYPO3 7.6 and above, you can add this to your AdditionalConfiguration.php:

if (\TYPO3\CMS\Core\Utility\GeneralUtility::getApplicationContext()->isProduction()) {
    $GLOBALS['TYPO3_CONF_VARS'] = array_replace_recursive($GLOBALS['TYPO3_CONF_VARS'],
        [
            'LOG' => [
                'writerConfiguration' => [
                    \TYPO3\CMS\Core\Log\LogLevel::DEBUG => \TYPO3\CMS\Core\Log\Writer\NullWriter::class,
                    \TYPO3\CMS\Core\Log\LogLevel::INFO => \TYPO3\CMS\Core\Log\Writer\NullWriter::class,
                    \TYPO3\CMS\Core\Log\LogLevel::NOTICE => \TYPO3\CMS\Core\Log\Writer\NullWriter::class,
                    \TYPO3\CMS\Core\Log\LogLevel::WARNING => \TYPO3\CMS\Core\Log\Writer\NullWriter::class
                ]
            ]
        ]
    );
}

or change the writer configuration just for the Realurl-Url-Encoder class:

$GLOBALS['TYPO3_CONF_VARS']['LOG']['DmitryDulepov']['Realurl']['Encoder']['UrlEncoder']['writerConfiguration'] = [
    \TYPO3\CMS\Core\Log\LogLevel::ERROR => [
        \TYPO3\CMS\Core\Log\Writer\NullWriter::class => [],
    ],
];

Additionally, if your side doesn't use multiple languages, you could also remove the L parameter from your TypoScript setup:

config.linkVars =

If you use other linkVars, of course you should leave them as before.

@leslawp
Copy link

leslawp commented Nov 13, 2017

Thank you, @mediaessenz !
I will try that!

@leslawp
Copy link

leslawp commented Nov 15, 2017

Looks like it works :)
Still we need to think about steady solution, since we are doing workarounds...

@neotrace
Copy link

I am also experiencing this problem in TYPO3 7.6. Server eventually ran out of space and crashed due to a 60GB Log file full of these RealURL Errors. I have L(0-6) set in my config since I only have those languages, but I see requests for L=10/11/12 and so on coming from outside. Probably Searchengine or bot trying languages.
Our Webhoster made a symlink to /dev/null for the log file so that is another working solution I wanted to share here. Waiting desperately for a fix.

@webian
Copy link
Contributor

webian commented Nov 29, 2017

The huge logs are generated from here: https://github.com/dmitryd/typo3-realurl/blob/development/Classes/Encoder/UrlEncoder.php#L1589
That debug_backtrace() returns an array with more or less 30 elements and each of them have inside the whole \DmitryDulepov\Realurl\Encoder\UrlEncoder object and are later json encoded. This is repeated for every link in the page generating a ~250MB file for each page.

@webian
Copy link
Contributor

webian commented Nov 30, 2017

I made this rewrite rule for .htaccess to remove unwanted L parameters:
RewriteCond %{QUERY_STRING} ^(.)&?L=(?!(1|11)(&|$))[^&]+&?(.)$
RewriteRule ^/?(.*)$ /$1?%1%4 [R=301,L]

Play with the regex: https://regex101.com/r/KsOYvR/1

In this case I allow only L=1 and L=11.
Change 1|11 with the ID that you want to allow, i.e. 2|4|5|15, all other values will cause the L parameter to be removed.

@osahner
Copy link

osahner commented Nov 30, 2017

#551 fixes the Logfile overflow.
I wonder how long we have to wait for a release ...

@fazzyx
Copy link

fazzyx commented Nov 30, 2017

@webian And what if u have partial translated content?

@webian
Copy link
Contributor

webian commented Nov 30, 2017

@fazzyx sorry I can't understand what you're asking. Can you explain your doubts further?

@fazzyx
Copy link

fazzyx commented Nov 30, 2017

@webian Basically you translate to another language, but not every content / site.

@webian
Copy link
Contributor

webian commented Nov 30, 2017

@fazzyx my rewrite rule should be used to remove L parameter with values not configured. Does your case have urls with values for L parameter not configured?
ie.: you use default language English (uid=0) and a second language german (uid=1). Then you change the regex to allow only L=0 or L=1: ^(.)&?L=(?!(0|1)(&|$))[^&]+&?(.)$.
Did I miss something?

@fazzyx
Copy link

fazzyx commented Nov 30, 2017

@webian I just described a scenario where not every content or page are translated. You have only a workaround more, but we need a solution.

@franzkugelmann
Copy link
Contributor

we were able to stop the logging with TypoScript
config.linkVars := addToList(L(0|1|4))
You need to add your language-ids instead of 0|1|4.
Then TYPO3 does not let the not-existing language id through and the logger in realurl is not triggered.

@adrienjacob
Copy link
Contributor

There's a typical issue with bootstrap_package : you need to disable unneeded languages in the template constants - page.theme.language.languageValue = 0 (if you only use the default language)

cf. this other thread :
#407 (comment)

@wtfred
Copy link

wtfred commented Jan 8, 2018

@outdoorsman Did you get the 2.3.1 from TYPO3 TER or directly from github ? I made the update from the TER but i'm still getting huge log files...

@sgrossberndt
Copy link
Contributor

2.3.1 does not contain the fix: 2c66992#commitcomment-26436691

@outdoorsman
Copy link

outdoorsman commented Jan 8, 2018

When I first updated it, it seemed to keep spewing log files that went in the the gigabytes within minutes. But if I remember right (and I'm not sure about this), I cleared cache from the TYPO3 backend and it didn't work but then cleared them from the Install Tool which I believe does it much more thoroughly and then it reduced down to single line entries instead of a full trace... at least manageable if you don't leave it too long. I used the Development branch from... https://github.com/dmitryd/typo3-realurl.git. You should be in Production/Live mode too.

@franzholz
Copy link
Contributor

franzholz commented Jan 8, 2018

I do not understand the wiki page
https://github.com/dmitryd/typo3-realurl/wiki/Notes-for-Integrators#i-see-a-lot-of-strange-entries-in-the-realurl-module
Why are these strange L parameters generated from anywhere inside of TYPO3? I thought all links are generated by realurl if activated. Therefore these links with wrong L parameters should not exist at all.

[08-Jan-2018 19:27:02 Europe/Berlin] http://mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("courier-authlib-0.59.1-1.x86_64.rpm") was detected by realurl. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!

If I use the TYPO3 backend to search for a term "courier-authlib-0.59.1-1.x86_64", then I have no result.
I have made a search on the filesystem. No file "courier-authlib-0.59.1-1.x86_64.rpm" exists. It has been removed many years ago.

@outdoorsman
Copy link

There's a typical issue with bootstrap_package : you need to disable unneeded languages in the template constants - page.theme.language.languageValue = 0 (if you only use the default language)

@adrienjacob Thanks for that tip about the above issue in bootstrap_package. Unfortunately the fix you mentioned only works for version 7.1.0 and above and I have bootstrap_package version 7.0.5. It's also in a complicated integration so I can't just update it quickly.

@dmitryd
Copy link
Owner

dmitryd commented Jan 11, 2018

@franzholz

I thought all links are generated by realurl if activated. Therefore these links with wrong L parameters should not exist at all.

No. Links are generated to TYPO3 and only passed to realurl to encode to speaking url. So if TYPO3 makes /index.php?id=123&L=test, realurl can only complain that it got invalid L.

@jean-alex
Copy link

The fix suggested by @adrienjacob solved my problem.

@daCyberpunk
Copy link

I'm using also the workaround mentioned here, but I'm not happy with. Because on our staging system we have the same problem and if we are using the workaround, then we can not set the Debug settings in Installtool to debug.
I also tried to set page.theme.language.languageValue = 0,1,6,9,10 and i tried config.linkVars = L(0-10) as well as config.linkVars = :=addToList(L(0|1|6|9|10)) but still we get the error about the bad L parameter (2).
What I can add to this discussion is, that we have in Backend added the french language which actually has the id 2 but we set it to hidden. I don't know if that could be the problem.

@franzholz
Copy link
Contributor

@dmitry: Now I did more investigations. I collected the IP addresses of the users who's visits to my website generate those masses of entries into the PHP error_log file by realurl:

[24-Jan-2018 09:29:30 Europe/Berlin] http://www.mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("links") was detected by realurl for IP 46.229.168.73. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!
[24-Jan-2018 09:29:30 Europe/Berlin] http://www.mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("links") was detected by realurl for IP 46.229.168.73. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!
[24-Jan-2018 09:29:48 Europe/Berlin] http://mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("seite-nicht-gefunden") was detected by realurl for IP 216.244.66.200. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!
[24-Jan-2018 09:29:48 Europe/Berlin] http://mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("seite-nicht-gefunden") was detected by realurl for IP 216.244.66.200. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!
[24-Jan-2018 09:29:48 Europe/Berlin] http://mywebsite.com/ - cms: $TSFE->set_no_cache() was triggered. Reason: Bad "L" parameter ("seite-nicht-gefunden") was detected by realurl for IP 216.244.66.200. Page caching is disabled to prevent spreading of wrong "L" value.. Caching is disabled!

Both IP addresses are listed as known for making attempts for DDoS attacks.

Do you still continue to think that TYPO3 generated those wrong L parameters?

@dmitry
Copy link

dmitry commented Jan 24, 2018

/cc @dmitryd

@co-operate
Copy link

Same behaviour here on several sites with TYPO3 8.7.9 + RealURL 2.3.1. Most notably on one site with basic.enableAutoConf=true and config.linkVars=L(0-1) set and in standard production envirnoment settings (including the standard 'SYS/systemLogLevel' => '2'). On this site, the log file entries seem to be triggered by sql injection attempts:

Sun, 28 Jan 2018 07:48:40 +0100 [ERROR] request="6fac06f4c348d" component="DmitryDulepov.Realurl.Encoder.UrlEncoder": Bad "L" parameter ("1 AND 1=2 UNION SELECT 0x6461726b31636f6465--\'A=0\'A=0") was detected by realurl. Page caching is disabled to prevent spreading of wrong "L" value

As a consequence, the log file currently grows by 2 GB/h. Each new line adding >20MB to the log file. I think that's not acceptable in a production environment...

@DiablosOffens
Copy link

I had the same impression in the past, but found the actual root cause of the problem on our website.
It wasn't a problem in typo3 core or the realurl extension. We use an extension which forces all requesters to redirect to URLs provided by realurl if the requested URL includes "index.php". I fixed the bug there and now all log messages are gone.
I provided this fix for the author's git repository.

The stack trace in the log messages were actually a real help to find this bug.

@outdoorsman
Copy link

I had the same impression in the past, but found the actual root cause of the problem on our website.
It wasn't a problem in typo3 core or the realurl extension. We use an extension which forces all requesters to redirect to URLs provided by realurl if the requested URL includes "index.php". I fixed the bug there and now all log messages are gone.
I provided this fix for the author's git repository.

Would it make sense for something like the forcerealurls2 to be integrated into realurl out of the gate? I see this extension still doesn't support TYPO3 8.7, which is a deal breaker for some of my sites.

@DiablosOffens
Copy link

Would it make sense for something like the forcerealurls2 to be integrated into realurl out of the gate? I see this extension still doesn't support TYPO3 8.7, which is a deal breaker for some of my sites.

Yeah sure, removing the need for such outdated extensions would also reduce the possible attack vectors.
But it has to be something like the fixed version I'm currently using.

@CDRO
Copy link

CDRO commented Feb 1, 2018

+1

@ribase
Copy link

ribase commented Feb 9, 2018

Is it possible to lockdown 2.3.1 and 2.3.0? At this point we have to set a fix version in every composer.json...

This is a serious bug and will "DDOS" the system itself...

@outdoorsman
Copy link

Is it possible to lockdown 2.3.1 and 2.3.0? At this point we have to set a fix version in every composer.json... This is a serious bug and will "DDOS" the system itself...

This has taken down an entire server full of websites multiple times. After some work I have been able to reduce the amount of logs but they are still filling up with imaginary errors as far as I can tell. The default should not be to blow up the system with logs like this, maybe you can add a debug checkbox mode to the extension or something but when in production this shouldn't happen!

@xperseguers
Copy link

Same problem here, logs are filling up my server disk constantly and very rapidly. Version 2.3.1 installed.

@No5251
Copy link

No5251 commented Mar 5, 2018

If you're using the "bootstrap_package", set correct values for
config.linkVars and page.theme.language.languageValue.

@dmitryd
Copy link
Owner

dmitryd commented Mar 6, 2018

@outdoorsman

This has taken down an entire server full of websites multiple times. After some work I have been able to reduce the amount of logs but they are still filling up with imaginary errors as far as I can tell. The default should not be to blow up the system with logs like this, maybe you can add a debug checkbox mode to the extension or something but when in production this shouldn't happen!

May be, you can configure your system correctly after all this? What do you think? ;) These are not imaginary errors but a faulty implementation of your site.

@dmitryd
Copy link
Owner

dmitryd commented Mar 6, 2018

@co-operate Same for you: configure the site correctly! Your site allows arbitrary L values. The one from logs shows that you allow values that contain SQL injections on your site.

@wtfred
Copy link

wtfred commented Mar 6, 2018

Same for you: configure the site correctly! Your site allows arbitrary L values. The one from logs shows that you allow values that contain SQL injections on your site.

Same behaviour here on several sites with TYPO3 8.7.9 + RealURL 2.3.1. Most notably on one site with basic.enableAutoConf=true and config.linkVars=L(0-1) set and in standard production envirnoment settings (including the standard 'SYS/systemLogLevel' => '2'). On this site, the log file entries seem to be triggered by sql injection attempts:
Sun, 28 Jan 2018 07:48:40 +0100 [ERROR] request="6fac06f4c348d" component="DmitryDulepov.Realurl.Encoder.UrlEncoder": Bad "L" parameter ("1 AND 1=2 UNION SELECT 0x6461726b31636f6465--\'A=0\'A=0") was detected by realurl. Page caching is disabled to prevent spreading of wrong "L" value

@dmitryd I thought config.linkVars=L(0-1) was the way to restrict L values to 0 and 1 only ? What's wrong with his configuration ?

@outdoorsman
Copy link

@dmitry as far as I can tell I've followed your advice on how to configure and still get errors. Don't know what to think, but I still say that I've used this extension for years and never had an issue until now, all of a sudden, things are blowing up. I didn't change anything on my end. Then I tried to follow the directions you gave and it though it did reduce things I still couldn't get it to stop slowing filling my logs. I love verbose logging when needed but I still think the default should not be something so verbose that it crashes servers. Please don't take it personally... I'm just sharing one opinion of what would be good practice. I will still use and be grateful for the extension either way.

@fazzyx
Copy link

fazzyx commented Mar 7, 2018

If I change the page content, maybe you drop a (multilanguage) page (but not the whole language), there are still entries anywhere outside
and if a bot calling these sites the log file get flooded.
How can I managed this, what is the right setup for this?

@dmitryd
Copy link
Owner

dmitryd commented Mar 7, 2018

@Anubiso
Copy link

Anubiso commented Mar 22, 2018

I have a single-language page and also get log entries. My Config looks like this:

TypoScript-Setup:

config{
   linkVars = L(0)
   sys_language_uid = 0
   language = de
}

realurlconf:

'preVars' => array(
    array(
        'GETvar' => 'L',
        'noMatch' => 'bypass',
    ),
)

TYPO3 doesn't generate the faulty links, but when i call the page with a faulty L parameter manually the log file is flooded. Worst case: Using a faulty L parameter on a paginated news list page where 25 log entries (250MB) are created on 1 request.

Example-URL valid:

id=113&tx_news_pi1%255B%2540widget_0%255D%255BcurrentPage%255D=2&cHash=ea27ba24a8da3abdbb255e329207e874

Flood the logs:

id=113&L=142&tx_news_pi1%255B%2540widget_0%255D%255BcurrentPage%255D=2&cHash=ea27ba24a8da3abdbb255e329207e874

@Anubiso
Copy link

Anubiso commented Mar 22, 2018

Additional Info:
I think the problem is a core problem as the UriBuilder used in the widget-link view-helper obviously doesn't validate the language parameter. (the wrong L parameter is used in all pagination links)
<f:widget.link addQueryStringMethod="GET"`

Nevertheless i think there should be an option to disable/shorten the log made by realurl, because there is no easy way to fix this.

@dmitryd
Copy link
Owner

dmitryd commented Apr 2, 2018

@Anubiso

TYPO3 doesn't generate the faulty links, but when i call the page with a faulty L parameter manually the log file is flooded.

Contradiction here. If log is flooded, it means that TYPO3 called realurl with a wrong L value. You can direct logs to nowhere.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests