Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Memory Consumption Problem in 2.1.0 #147

Closed
SegFaulty opened this issue Mar 24, 2020 · 1 comment
Closed

Memory Consumption Problem in 2.1.0 #147

SegFaulty opened this issue Mar 24, 2020 · 1 comment

Comments

@SegFaulty
Copy link

SegFaulty commented Mar 24, 2020

after upgrading today to 2.1.0 (on ubuntu 16.04)
We noticed a permanently increasing memory consumption in the run.
This resulted in many cases in out of memory errors.
We had to switch back to version 2.0.3 (We tested 2.0.1, this version was fine too)

We are streaming and decoding from a testfile with 600MB and 2Mio Messages.
Version 2.0.1 and 2.0.3 needed 4-6 MB php-memory for streaming through the whole file.
Version 2.1.0 increases memory with every message until its fails at 750MB memory limit!

php -i of failure version:
msgpack
extension Version => 2.1.0
header Version => 3.2.0

We used execute, unpack, reset ... see code below:

protected function getMessages__php_msgpack($messageCount = 1){
	$messages = array();
	if( $this->filePointer ){
		$continue = true;
		while( $continue ){
			$offset = 0;
			$this->unpacker->reset();
			$executeResult = $this->unpacker->execute($this->buffer, $offset);
			// since msgp-version 0.5.8 execute is true even if the msg-pack string is incomplete, but then offset is greater then buffer-size
			if( $executeResult AND $offset<=strlen($this->buffer) ){ 
				$this->buffer = substr($this->buffer, $offset);
				$messages[] = $this->unpacker->data();
				if( count($messages)==$messageCount ){
					$continue = false;
				}
			} else{
				// buffer not decoded -> feed more
				if( !feof($this->filePointer) ){
					// feed the buffer
					$this->buffer .= fread($this->filePointer, $this->readBufferSize);
				} else{
					// end of file
					$continue = false;
					$this->endOfFile = true;
					if( $this->buffer!='' ){
						trigger_error(__METHOD__.' buffer not empty at eof:'.Waps_Helper_String::getStringRepresentationOfBinary($this->getBuffer()), E_USER_WARNING);
						$this->errorOccurred = true;
					}
				}
			}
		}
	}
	return $messages;
}
@m6w6
Copy link
Collaborator

m6w6 commented Mar 28, 2020

Thank you for your report!

Could you possibly provide a - maybe obfuscated - example message?

@m6w6 m6w6 closed this as completed in f933cf5 Jul 31, 2020
m6w6 added a commit that referenced this issue Nov 27, 2020
* PHP-8 compatibility. (Remi Collet, Michael Wallner)
* Fixed #152: Split tests for #67 regarding 32/64bit platforms. (Andy Postnikov)
* Fixed #153 and #147: Small string optimization causes excessive memory
  consumption under specific workloads. (Michael Wallner)
m6w6 added a commit that referenced this issue Nov 27, 2020
* PHP-8 compatibility. (Remi Collet, Michael Wallner)
* Fixed #152: Split tests for #67 regarding 32/64bit platforms. (Andy Postnikov)
* Fixed #153 and #147: Small string optimization causes excessive memory
  consumption under specific workloads. (Michael Wallner)
m6w6 added a commit that referenced this issue Aug 20, 2024
m6w6 added a commit that referenced this issue Aug 20, 2024
* PHP-8 compatibility. (Remi Collet, Michael Wallner)
* Fixed #152: Split tests for #67 regarding 32/64bit platforms. (Andy Postnikov)
* Fixed #153 and #147: Small string optimization causes excessive memory
  consumption under specific workloads. (Michael Wallner)
m6w6 added a commit that referenced this issue Aug 20, 2024
* PHP-8 compatibility. (Remi Collet, Michael Wallner)
* Fixed #152: Split tests for #67 regarding 32/64bit platforms. (Andy Postnikov)
* Fixed #153 and #147: Small string optimization causes excessive memory
  consumption under specific workloads. (Michael Wallner)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants