Loki no longer runs on 32bit architecture #5388
Comments
|
We should be able to |
|
👋 sorry for randomly dropping in :) The problem does originate from the protobuf, but it might be helpful to distinguish it not as a protobuf problem, but rather a problem of the Go structs which are generated from the protobuf. (i.e., if gogoproto generated aligned structs with padding, you'd have no issue here). There's a few things you can do:
Adding an unused padding field so the generated Go structs are aligned: message Ingester {
// Total ingester reached for this query.
int32 totalReached = 1 [(gogoproto.jsontag) = "totalReached"];
int32 unusedTotalReachedPadding = 6;
// Total of chunks matched by the query from ingesters
int64 totalChunksMatched = 2 [(gogoproto.jsontag) = "totalChunksMatched"];
// Total of batches sent from ingesters.
int64 totalBatches = 3 [(gogoproto.jsontag) = "totalBatches"];
// Total lines sent by ingesters.
int64 totalLinesSent = 4 [(gogoproto.jsontag) = "totalLinesSent"];
Store store = 5 [(gogoproto.nullable) = false, (gogoproto.jsontag) = "store"];
} |
|
Very interesting idea @rfratto in fact I wonder if it could be even easier to just reorder the structs without adding a padding variable? I think however the bigger question is: is it even worth it anymore. Without a way to lint for this problem it feels like we'll always be chasing our tails here and I wonder if it's just time to throw in the towel and say 32bit doesn't work anymore. |
Yeah, it'd probably be hard to have a linting rule which isn't overaggressive but still catches most of the alignment problems that'd you'd care about |
|
Hi! This issue has been automatically marked as stale because it has not had any We use a stalebot among other tools to help manage the state of issues in this project. Stalebots are also emotionless and cruel and can close issues which are still very relevant. If this issue is important to you, please add a comment to keep it open. More importantly, please add a thumbs-up to the original issue entry. We regularly sort for closed issues which have a We may also:
We are doing our best to respond, organize, and prioritize all issues but it can be a challenging task, |
|
Hey, I actually have this problem on my Raspberrypi 4. I'm using docker with the image |
I tried the 2.4.0 on my Raspberry 3b+ and it works |
|
The 3b+ and 4 both support 64bit operating systems and the Raspberry Pi OS 64bit OS is officially released now. So as long as you run the 64bit OS you should have no problem running Loki on a 3b+ or 4. |
|
Any way to debug this? I explicitly tried |
|
Make sure your OS is 64bit: (If it's 32bit it will say If it's not |
|
|
I'm not sure then? that looks like the correct arch. It should just work even without specifying the arch on the image, docker will choose the correct architecture. I know Loki runs on a pi4 with 64bit os. I have several running myself. If you are seeing an 'unaligned 64bit operation' error that really feels like something is still 32bit... I know the arm chips will support running as 32 and 64 so maybe make sure the docker images downloaded are all cleared out? Maybe download the 64bit binary and try running it directly 69 narrow it down to a docker vs OS issue? |
|
I went into the container and executed the same command: |
|
I'm facing this problem when trying to run Loki 2.6.1 on 32-bit ARM architecture (armv7l) running on Synology NAS. If I understand this is not going to be fixed in future release, right? Will you officially obsolete 32-bit ARM platform? |
This is true ? |
|
This is a particularly painful problem for me. I recently started using Loki to manage logs on a fleet of 32-bit arm devices, and ran into this problem when trying to upgrade my test devices from 2.4.2-arm to 2.6.1-arm. In my personal experience developing golang applications for ARM that use protobuf, it is better to use a go defined type as much as possible throughout the application, and convert to the generated protobuf message before sending across the wire. This allows more flexibility and control within the application. In this case it would allow for proper alignment for atomic operations by either padding or re-ordering the fields. gogoproto has a face plugin that may be useful here, as it would generate the code to convert to the proto message as long as the getter interface matches. This would also help enforce that the generated code and go code stay in sync. |
|
does anyone have a comment on @adferm approach here? from the sounds of it, the code is somehow using protobof as a value holder on the stack instead of a transport? if so, from my past experience programming in protobuf, I agree as well. |
|
If anything, can you please remove the loki-linux-arm.zip (arm 32) builds as that is obviously broken now. |
|
thanks for this information. I stumbled over the same on my raspberry pi 2 and went back to vesion 2.4.0 which is working. thanks |
slim-bean
added
the
keepalive
This is mostly for posterity because I don't think we will likely fix this.☹️
I think the problem starts here:
I don't totally understand what's happening here but this isn't the first time I've seen an offset problem in Loki trying to run on 32bit architecture.
The object being added, and the struct it's being added it to, don't seem to be the problem directly it's a 64bit variable in a struct where everything is 64bit.
I think the problem is this struct is included in another struct with other structs, one of which has a 32bit variable:
loki/pkg/logqlmodel/stats/stats.proto
Line 45 in 191a8a6
If this is in fact the problem, I think there is a bigger problem. I don't know how to fix this without making a breaking protobuf change. We need basically to make that
totalReached64 bit or remove it and put a 64bit version of it at the end.Pretty sure making it 64bit would be a breaking change, I think we would have to remove it and add a new var at the end which is 64bit. But I think for this to be not breaking it would need to be deprecated for a few versions and removed.
If I'm right and if this works, Is it worth it? This takes time, it's hard to test (I have one raspberry pi 3b+ i'm trying to run Loki on and it's probably gonna be running a 64bit OS in a few hours since finding this)
What else is gonna be broken, I'm updating this Pi after 2 years, I have no idea where else there may be problems and I'm sorry to say without someone else who would really want to champion this I think 32bit support has ended....
If there was a way to lint for this problem and we could scope the work to fix what's currently broken and most importantly be able to find new bugs when they are introduced, then I think we could continue support for 32bit.
The text was updated successfully, but these errors were encountered: