Cannot create more than 65536 PoolByteArrays

[path9263]

Godot version:
3.0.1

OS/device including version:
Windows 7, 64bit

Issue description:
Attempted to create a very large number of PoolByteArrays but as soon as more than 65536 (maximum for a 16 bit number) of them are created the game becomes unresponsive and crashes.

Steps to reproduce:
The following code crashes but if you change the variable maxNum to 65535 or less it works as expected.

extends Node

var testArr = []
var randNumber
var randNumberTwo
var maxNum = 65536  # <- reduce by one and this code works

func _ready():
	for i in maxNum:
		randNumber = randi() % 100
		randNumberTwo = randi() % 200
		var bytePool = PoolByteArray()
		bytePool.push_back(randNumber)
		bytePool.push_back(randNumberTwo)
		testArr.push_back(bytePool)

	print("Done!")
	print(testArr.size())

Minimal reproduction project:
PoolByteArrays.zip

[eon-s]

I did this:

extends Node

var testArr = []
var randNumber
var randNumberTwo
var maxNum = 65535  # <- reduce by one and this code works

func _ready():
	create_arrays()
	call_deferred("create_arrays")

func create_arrays():
	for i in maxNum:
		randNumber = randi() % 100
		randNumberTwo = randi() % 200
		var bytePool = PoolByteArray()
		bytePool.push_back(randNumber)
		bytePool.push_back(randNumberTwo)
		testArr.push_back(bytePool)

	print("Done!")
	print(testArr.size())

and got on output:

Done!
65535
Done!
131070

and a couple of errors:

0:00:00:0627 - All memory pool allocations are in use.
----------
Type:Error
Description: All memory pool allocations are in use.
Time: 0:00:00:0627
C Error: Method/Function Failed, returning: ERR_OUT_OF_MEMORY
C Source: core/dvector.h:524
C Function: resize

ending with:

0:00:27:0167 - Too many errors! 262126 errors were dropped.
----------
Type:Error
Description: Too many errors! 262126 errors were dropped.
Time: 0:00:27:0167
C Error: TOO_MANY_ERRORS
C Source: :1795354114
C Function: 

[qarmin]

When I run a game with code from above, this error appear(really strange):

[1] /lib/x86_64-linux-gnu/libc.so.6(+0x3ef20) [0x7f48a8822f20] (??:0)
[2] ProceduralSky::_generate_sky() (/home/rafal/Pulpit/mojgodot/scene/resources/sky.cpp:217 (discriminator 2))
[3] ProceduralSky::_update_sky() (/home/rafal/Pulpit/mojgodot/scene/resources/sky.cpp:415)
[4] MethodBind0::call(Object*, Variant const**, int, Variant::CallError&) (/home/rafal/Pulpit/mojgodot/./core/method_bind.gen.inc:56 (discriminator 4))
[5] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (/home/rafal/Pulpit/mojgodot/core/object.cpp:945 (discriminator 1))
[6] MessageQueue::_call_function(Object*, StringName const&, Variant const*, int, bool) (/home/rafal/Pulpit/mojgodot/core/message_queue.cpp:256)
[7] MessageQueue::flush() (/home/rafal/Pulpit/mojgodot/core/message_queue.cpp:305)
[8] SceneTree::iteration(float) (/home/rafal/Pulpit/mojgodot/scene/main/scene_tree.cpp:477 (discriminator 2))
[9] Main::iteration() (/home/rafal/Pulpit/mojgodot/main/main.cpp:1853)
[10] OS_X11::run() (/home/rafal/Pulpit/mojgodot/platform/x11/os_x11.cpp:3004)
[11] /home/rafal/Pulpit/mojgodot/bin/godot.x11.tools.64(main+0xdc) [0x115b5e3] (/home/rafal/Pulpit/mojgodot/platform/x11/godot_x11.cpp:56)
[12] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xe7) [0x7f48a8805b97] (??:0)
[13] /home/rafal/Pulpit/mojgodot/bin/godot.x11.tools.64(_start+0x2a) [0x115b44a] (??:?)

In my project, I read from file, with encoding other than utf-8, many thousands of characters which I must load with get_8() function and store it in PoolByteArray, and then I can easily overflow it.

[Zylann]

Godot 3.2.2 beta3
I might have stumbled upon this problem as well. In my plugin I paint images, and backup tiles of the image that were modified in order to use undo/redo. Turns out it's hitting Godot's limits faster than I thought, even though it occupies only 400 Mb and gigabytes are still available.

[Calinou]

Can someone try to reproduce this with PackedByteArray in the master branch?

[Anutrix]

Doesn't crash in master build 3f7f4fb.
Tested with:

extends Node2D

var testArr = []
var randNumber
var randNumberTwo
var maxNum = 65535  # <- reduce by one and this code works

func _ready():
	create_arrays()
	call_deferred("create_arrays")

func create_arrays():
	for i in maxNum:
		randNumber = randi() % 100
		randNumberTwo = randi() % 200
		var bytePacked = PackedByteArray()
		bytePacked.push_back(randNumber)
		bytePacked.push_back(randNumberTwo)
		testArr.push_back(bytePacked)

	print("Done!")
	print(testArr.size())

Which results in:

Done!
65535
Done!
131070

Update:
Inspecting/Clicking the node having the above script in Remote Tree crashes the running game but not the editor itself.

[TheSofox]

Did a test with a hundred times the original number, it took maybe 10 seconds to do, but completed fine:

Godot v4.3.dev (0c03d25) - Windows 10.0.19045 - Vulkan (Forward+) - dedicated NVIDIA GeForce GTX 1070 (NVIDIA; 31.0.15.3623) - Intel(R) Core(TM) i7-7700K CPU @ 4.20GHz (8 Threads)

[AThousandShips]

This doesn't guarantee that it work on 3.x as the code is largely different, so to properly close this testing on 3.x would be required, but confirming it's not occuring on the 4.x branch is good!

[TheSofox]

I reproduced it from the LTS binary from the website ( v3.5.3.stable.official [6c81413] ).
Highest value I was able to get was 65532. Any higher and it crashed.