[Questions] Generic server terminating - "badmatch,{error,eacces}" error from rabbit_classic_queue_store_v2 #14094
Replies: 3 comments 21 replies
-
|
@Donderstal this is a known Windows-specific error: sometimes Windows returns We will see in the next patch release or two. |
Beta Was this translation helpful? Give feedback.
6 replies
-
|
This error is for classic queues not quorum queues -- perhaps @lhoguin has some ideas. I've never seen |
Beta Was this translation helpful? Give feedback.
-
|
Using |
Beta Was this translation helpful? Give feedback.
-
|
Just now I reinstalled RabbitMq to the problematic server. If the problem still persists, I wil share the file/folder structure and and a ProcessMonitor trace as you requested. If so I will update you later today. |
Beta Was this translation helpful? Give feedback.
-
|
@Donderstal I doubt that reinstalling RabbitMQ will resolve anything. This is a Windows file handle API behavior that's different from other OSes. It could also be security software side effects, so if you run any on the same hosts, please whitelist all RabbitMQ processes, data directories, and so on. Or even disable the anti-virus tools, if your nodes are isolated from the public Internet (as they should be). |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin I've contacted the sysadmin of this customers' network and they have guaranteed me that all RabbitMQ processes are whitelisted. The nodes are also isolated from public internet. Both were already the case before I made a post here. I just figured I could try a reinstall, since the problem is quite troublesome for this customer and I didn't have any better ideas right now. The reinstall seems to have sharply decreased the frequency of the errors to only once today, but this could just be a coincidence. I've used ProcessMonitor to capture the file system reads/writes this afternoon. I've limited it to just the erl.exe while monitoring. During my monitoring today, the error didn't occur. I can share the Logfile.PML and the RabbitMq file/folder structure, but I figured it's probably useless to you all because it doesn't capture the moment of the Monday I will try running the ProcessMonitor all day until I successfully capture the error. When I do I will share it here. Thank you all very much for your help. |
Beta Was this translation helpful? Give feedback.
-
|
This error occured just now while I was running Process Monitor. This is the error in the RabbitMq logs: I've added the ProcessMonitor logfile as .CSV as attachment. I also have a full .PML file available but I can't add .PML files into a github post. I could email it if you'd like. |
Beta Was this translation helpful? Give feedback.
15 replies
-
|
@Donderstal You can try an alpha build (of future 4.2.0) available at https://github.com/rabbitmq/server-packages/releases/tag/alphas.1750838043824 - upgrading should be smooth but make sure to backup your data. |
Beta Was this translation helpful? Give feedback.
-
|
We can also provide a |
Beta Was this translation helpful? Give feedback.
-
|
That is great news guys, thank you very much! If it's not too much trouble I'd prefer a 4.1.2 preview. I'm a bit anxious to deploy an alpha build to a customers' production environment. |
Beta Was this translation helpful? Give feedback.
-
|
#14131 is merged, I will share a link to a |
Beta Was this translation helpful? Give feedback.
-
|
@Donderstal this |
Beta Was this translation helpful? Give feedback.
-
|
@michaelklishin @lhoguin |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Community Support Policy
RabbitMQ version used
4.1.0
Erlang version used
27.3.x
Operating system (distribution) used
Windows Server 2016
How is RabbitMQ deployed?
Windows installer
rabbitmq-diagnostics status output
See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics
Logs from node 1 (with sensitive values edited out)
See https://www.rabbitmq.com/docs/logging to learn how to collect logs
Logs from node 2 (if applicable, with sensitive values edited out)
No response
Logs from node 3 (if applicable, with sensitive values edited out)
No response
rabbitmq.conf
I do not have one.
Steps to deploy RabbitMQ cluster
Steps to reproduce the behavior in question
I am unsure what causes this behavior. It happens intermittantly. Other servers with the exact same RabbitMQ/Erlang combination do not have this problem.
advanced.config
See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location
Application code
No response
Kubernetes deployment file
No response
What problem are you trying to solve?
I work at a software company on a on-premise solution that uses RabbitMQ as a message broker between different services on a application server. After updating to RabbitMQ 4.1.0, Erlang 27.3.3, some queues on one specific customer server crash, drop messages and restart. This can happen to any queue experiencing relatively high traffic. When it does, we get the error message in the RabbitMQ logs that I have shared above. Occassionally this leads to a complete failure of RabbitMQ, where no messages get published up at all anymore. When this happens, the RabbitMQ service is still running and we can still reach Rabbit's localhost management console. After restarting the RabbitMQ service, everything runs normally for a while before the whole issue starts over again.
I have spent some time trying to solve this, but so far I am unable to find a clear cause or solution. I've tried the following things:
I want to note that this only happens on one specific customers' server. I guess this is probably some kind of permission issue regarding the buffer files (?), but am unsure how to reproduce the issue or how I could attempt to solve it. If anyone could give some advice on what to do, that'd be much appreciated.
The error message points to the following line of code:
Beta Was this translation helpful? Give feedback.
All reactions