ErrorException: Allowed memory size: Full export of database

[kumy]

Calling an "export type 1" for long duration break on over memory consumption.

https://geokrety.org/export.php?modifiedsince=20000510032102&kocham_kaczynskiego=********

https://sentry.kumy.org/geokrety/geokrety-legacy/issues/547/

ErrorException: Allowed memory size of 134217728 bytes exhausted (tried to allocate 123015360 bytes)
  File "db.php", line 191, in handleFatalError
    $result = $this->mysqli->query($sql);
  File "templates/sentry-php-master/lib/Raven/ErrorHandler.php", line 0
    Raven_ErrorHandler->handleFatalError

[romain-dartigues]

By looking at

geokrety-website/db.php

Line 191 in 757f300

$result = $this->mysqli->query($sql);

I suspect it's PHP trying to be smart, where "buffered queries are the default", which is at at fault.

Try with unbuffered queries / mysqli_use_result.

[boly38]

Anyway, buffer or not, and out of error, I dont see any kind of limit onto the sql query. That's sounds like bad practice, and security issue.

[kumy]

@boly38 this script is for exporting the whole database, so, why "not using LIMIT" is a security risk?

[boly38]

DoS of someone querying export without constraints.

[boly38]

Another way to fix it: adding rules to prevent more than one export per IP. per time unit.
Eg. 1 export per 8 hour for anonymous. 1 export per hour for partners

[kumy]

DoS of someone querying export without constraints.

Without token, requests are limited to 10 days.

adding rules to prevent more than one export per IP.

Nice idea. Could this be achieved using rate limit on nginx side?

[boly38]

yeah ..seems possible
https://www.nginx.com/blog/rate-limiting-nginx/

[kumy]

New dedicated issue for discussing rate limiting. #12