Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of memory with Guzzle #30

Closed
Jellyfrog opened this issue May 12, 2020 · 3 comments
Closed

Out of memory with Guzzle #30

Jellyfrog opened this issue May 12, 2020 · 3 comments

Comments

@Jellyfrog
Copy link

Jellyfrog commented May 12, 2020

A bit surprising this will allocate a lot of memory with large json files, in this example a 181MB one (Found here; https://github.com/zemirco/sf-city-lots-json/blob/master/citylots.json)

<?php

require_once __DIR__ . '/vendor/autoload.php';

$client = new \GuzzleHttp\Client();
$response = $client->request('GET', 'http://127.0.0.1:8001/storage/citylots.json');

// Gets PHP stream resource from Guzzle stream
$phpStream = \GuzzleHttp\Psr7\StreamWrapper::getResource($response->getBody());

foreach (\JsonMachine\JsonMachine::fromStream($phpStream) as $key => $value) {
  //
}
% php memory.php 
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /tmp/test/vendor/halaxa/json-machine/src/Parser.php on line 177
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 20480 bytes) in /tmp/test/vendor/guzzlehttp/promises/src/TaskQueue.php on line 24

I'm 100% guessing its due to json-machine is not registering as the sink for Guzzle
http://docs.guzzlephp.org/en/stable/request-options.html#sink

@Jellyfrog
Copy link
Author

Actually, the same for HttpClient;

<?php

use JsonMachine\JsonMachine;
use Symfony\Component\HttpClient\HttpClient;
use Symfony\Contracts\HttpClient\ResponseStreamInterface;

require_once __DIR__ . '/vendor/autoload.php';

function httpClientChunks(ResponseStreamInterface $responseStream)
{
    foreach ($responseStream as $chunk) {
        yield $chunk->getContent();
    }
}

$client = HttpClient::create();
$response = $client->request('GET', 'http://127.0.0.1:8001/storage/citylots.json');
$jsonChunks = httpClientChunks($client->stream($response));
foreach (JsonMachine::fromIterable($jsonChunks) as $key => $value) {
    //
}
% php memory.php 
PHP Fatal error:  Allowed memory size of 268435456 bytes exhausted (tried to allocate 132120592 bytes) in /tmp/test/vendor/halaxa/json-machine/src/Parser.php on line 108
* Closing connection 0

@halaxa
Copy link
Owner

halaxa commented May 12, 2020

You want to iterate features subtree, not the main level, right? If so, see the doc

@Jellyfrog
Copy link
Author

Oh, actually I didn't even look at the data, I just googled "large json" to find some test-data.
subtree seems to solve the issue.
Also, if changing to $response = $client->request('GET', 'http://127.0.0.1:8001/storage/citylots.json', ['stream' => true]);, then no temp-file is written to disk.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants