move2map FAILED?

[tomas]

Hi, I'm seeing a weird error when running the 0.8 branch in production (x64, Ruby 3.4.1).

ERROR:    move2map FAILED (0/8)!
SECURITY: hash map fio___http_hmap_s under attack? (partial/full collision guard) - capa: 4.
ERROR:    Couldn't add value to header: x-permitted-cross-domain-policies:none

And also:

SECURITY: hash map fio___http_hmap_s under attack? (partial/full collision guard) - capa: 1.
ERROR:    Couldn't add value to header: x-frame-options:SAMEORIGIN

I'm not sure why this might be happening. I don't recall seeing this in Iodine v0.7.

Strangely, some iodine worker processes seem to be stuck or frozen (ie. when sending a USR1 signal to the master process, they don't stop, and sending them a direct TERM signal doesn't work either). I'm not sure if this is related or caused by the errors mentioned above, but it might help to figure out what's going on.

Thanks again!

[boazsegev]

Hi @tomas ,

Thank you for bringing this to my attention.

It would really help me to hunt down the issue if you could send me an example request or an example response that causes this issue to occur.

The SECURITY log messages are designed to catch full hash collisions as well as excessive partial hash collisions. They really shouldn't happen with a single object unless somehow a full collision occurs.

Specifically, the partial/full collision guard message is logged when a collision is detected during re-hashing in a way that prevents the existing map (+ the new element) from fitting inside a newly allocated map with double the resources.

Some Hash Map implementations would ignore the issue, which could either result in additional memory usage (when also comparing actual key value) or data loss (when comparing only hashed value).

I'll try to see if I can discover why this might be happening. It's possible that there's a header name hash collision... possibly a significant partial collision... but it shouldn't raise any issues with such small map capacities.

[boazsegev]

Hi @tomas ,

Digging I found that the issue may have been related to the way I was testing if two objects are of the same type. I think Ruby 3.x changed the logic behind the value of a Ruby VALUE, so this may have broken some assumptions I had.

I hope the patch I committed will solve the issue. Please let me know.

Thanks,
Bo.

[tomas]

Hi @boazsegev, and thanks for the quick fix!

I had to revert to the previous Iodine version to keep things running so I'll have a look at this later when I have a minute. Do you think that might have been the reason why the worker processes stopped responding?

[boazsegev]

Hi @tomas,

I'm not sure why the worker processes stopped responding, but it's possible.

IMHO, I suspect that there is a busy loop somewhere that starves the reactor and prevents signal events from being handled.

In response, I updated the signal handling logic so it doesn't defer the signal handling (allowing a double SIGINT to always crash the system, no matter what user code iodine is running).

[tomas]

Great, I'll try to replicate the issue when I can. However, I don't think the processes stopped responding because/when they received a TERM or INT signal. I think they stopped responding far earlier (maybe because of the move2map bug), because the master process had already spawned a new set of processes and those were still hanging around.