Error when establishing the mqtt connection

[alexmaron81]

Hello @RobMeades .

I use an ESP32 with the ESP-IDF library and try again to establish a connection to the mqtt service.
And I get the following error message as shown in the picture.
But what I noticed is that if I use an earlier version of ubxlib, is the error not ther

e?

[RobMeades]

Hi there: looks like we're getting something back from the module which is somehow overrunning the end of the AT command buffer. Some questions:

• Can you indicate which version works and which version does not?
• I guess this is with your SARA-R412M module: can you confirm the full module name or, maybe paste in here the response to the ATI command which we issue at every power up (which will indicate the module firmware version)?
• Could you send an AT log including the MQTT-related AT exchange leading up to this (obviously obscuring any passwords etc.)

[alexmaron81]

On August 17th you made a pre-release version available in the case of "Unable to connect to MQTT broker # 26". This version works.

Yes it is the SARA-R412M 02B modu
V3_in_one_Task.txt
le

[alexmaron81]

but sometimes i get this error too.
ERROR A stack overflow in task eventTask has been detected.
Does this error also have something to do with it?

[RobMeades]

Hmm, yes, possibly. The abort that is going off is because of some guards we put on either end of the buffer that the incoming AT stream is placed in. Whenever that buffer is accessed we check that the values we wrote to the guard areas at the outset are unchanged; the assert is saying that something has written to them.

I've never seen this assert go off so I think it is relatively unlikely that there is an actual buffer overrun here; an out-of-bounds memory issue elsewhere may well be the cause.

I guess your code is not public but are you using any of the callback-type functions of the ubxlib API? These would be called in the context of an event task and so, if they happened to be using stack variables, there might be a need to increase the event task stack.

On the ubxlib versions, the pre-release I made available here would have been pretty much the same as that currently released, as far as MQTT is concerned, so there's probably not much of a clue for us there.

Is the problem nice and repeatable?

[alexmaron81]

I have increased the value "U_AT_CLIENT_URC_TASK_STACK_SIZE_BYTES" in the "u_at_client.h", now no more errors appear.

[RobMeades]

While that's good news I can't see how the URC task stack size would have any bearing on this and, of course, moving memory around will move the target; might make things work as a side-effect. Is the original problem quite reproducible? If so it might be better to see if we can think of a way to trap the problem.

[alexmaron81]

Yes, the error keeps coming back.
But when I use the earlier version I get a different output of the error. See file V2. There is a reference to the "eventTask" task and that's why I decided to increase the stack.
After increasing the stack and running my software, I saw the following. In the output "Unable to subscribe to topic ...." the IMEI number is shown and not the topic name. See file "V2_increased_URC_task"
V2.txt
V3_increased_URC_task.txt

[alexmaron81]

The latter with the topic is my mistake, I changed something in the example code.

[alexmaron81]

how do you calculate the stack size? do you have a tool for?

[alexmaron81]

so after I execute the code again as in the example and increase the stack size as described above, do I get another stack error?
V3_stack_increased.txt

I don't understand what I'm doing wrong and don't know where to look?

[RobMeades]

What I would do at this point is to instrument the code with some debug. Assuming we are still on the problem of hitting that buffer check-zone and assuming it is repeatable:

I would create a global variable in your main source file and call it int32_t gAlex[10], i.e. an array of ten int32_ts with some obviously unused global name; you can increase the size of the array as necessary but not too much as adding a global variable will shift memory about and that might affect the problem. Then, near the top of u_cell_mqtt.c, just underneath the #includes, extern it, i.e. add the line extern int32_t gAlex[];. Now you can populate that variable at the interesting points throughout the MQTT URC processing.

You will be able to see from your backtrace that the MQTT URC starts with UUMQTT_urc() and eventually goes "bang" some time after u_cell_mqtt.c line 261.

The function to obtain stack size you can find in u_port_os.h: uPortTaskStackMinFree(); just call it with NULL as the parameter and it will give you the stack high-watermark for the current task. You need to add calls to this function, storing the answer in your global variable, at the interesting points during the trail from UUMQTT_urc() to u_cell_mqtt.c line 261 according to your backtrace; for instance at the entry point of UUMQTT_urc() you could add the line gAlex[0] = uPortTaskStackMinFree(NULL);, and anywhere else along the line you think might be interesting, maybe at the entry points of the other URC handling functions (gAlex[1] = uPortTaskStackMinFree(NULL);, etc.), even down into uAtClientSkipParameters() and beyond (obviously externing your global variable near the top of u_at_client.c for this case). This will track the state of the URC task stack in the run-up to the problem.

Now, to get visibility of the values, open u_at_client.c and add the same extern int32_t gAlex[]; just below the #includes at the top if you haven't done so already. Then, find the places where it calls the guard check, i.e. assert(U_AT_CLIENT_GUARD_CHECK(pReceiveBuffer)); and replace each of those with something like:

if (!U_AT_CLIENT_GUARD_CHECK(pReceiveBuffer)) {
    uLogPrintf ("### Guard has been hit!\n");
    for (size_t alex = 0; alex < 10; alex++) {
        uLogPrintf ("### %d ", gAlex[alex]);
    }
    uLogPrintf ("\n");
}

Now, when the guard is hit, the stack left available to the URC task in the run-up to the point where we think it might have been breached should be printed out.

This is all a bit laborious of course but since the ESP32 has no debugger (that I know of) we have to do things manually like this. Note that I'm not suggesting we print the values out as we go as that would be too intrusive in terms of the behaviour of the code, would likely cause its own problems; instead collect them in an array and only print them out at the end.

Above you mention variants of the problem, where it might not be the AT buffer guard areas that are being hit but instead the stack is being flagged as being overrun. You can use the same approach in those cases: create a global variable you can populate with information in one file, extern and populate it in the interesting places and print it out at a point near that where the problem is flagged. For stack extent breaches, you might be best off increasing the offending stack size first, populating your debug variable with the value of uPortTaskStackMinFree(NULL) within code that is run in that task's context, and printing out your debug variable at a point which is called at regular intervals: you can then check the numbers to determine whether the stack really was too little, and be happy that increasing it was the right thing to do.

In your last example above you appear to be running the MQTT example we provide, which of course we run every time we touch the code in our automated test system, so I'm struggling to understand why a stack breach could be occurring unless there is something different in the build metadata (e.g. the code optimisation settings?).

[alexmaron81]

I'm not sure I did it right, but I got the output "printf (" StackSize =% d \ n ", uPortTaskStackMinFree (NULL));" into the file "u_cell_mqtt.c" after "uAtClientResponseStart (atHandle," + UMQTTC: ");" set and get "StackSize = 3188" output.

[alexmaron81]

therefore i increased the stack size for the task.
then I got another error message for another stack, so I increased another stack size.
I have currently set the two "U_AT_CLIENT_URC_TASK_STACK_SIZE_BYTES" and "U_AT_CLIENT_CALLBACK_TASK_STACK_SIZE_BYTES" to 5000 as a test.
so far my code runs smoothly.

Is there a general problem with choosing a stack larger than necessary?

[RobMeades]

Good stuff: remember that the number you are printing out there is the minimum stack free, ever, for that particular task, the "high watermark" of stack usage, so in the case you mention above there were 3188 bytes free on the stack of that task, which is quite a lot, so I think it is relatively unlikely to really be a stack problem in this case.

There is no down-side to choosing a larger stack size, other than being inefficient in RAM terms, but the ESP32 has quite a lot of RAM so this may not be an issue for you. The bigger issue is that this hasn't fixed whatever the problem is, it has simply moved RAM around so that the problem is no longer hitting a memory area you care about. Murphy's law says the problem will return!