Crash of C++ esminiLib in combination with FollowRouteController

[MichaelHarth]

Hello,

I am trying to use the C++ esminiLib with a scenario using the FollowRouteController. As soon as the controller tries to make a lane change (indicated by the comand line output: "1.328: LaneChange standbyState -> startTransition -> runningState"), the esminiLib crashes while executing the command SE_StepDT(0.001).
I have also tried to run the follow_route_with_lane_change.xosc from the examples, but I get the same issue here. However, running both scenarios without the esminiLib, but using the esmini executable works perfectly for me.

Can you reproduce my issue? And do you know, how to solve it?

Best regards,
Michael

[eknabevcc]

Thanks for reporting this issue! The root cause is that the target lane (road 0, lane -4) has width = 0.0 at the exact point where the lane change is being triggered. Then the lane width increases slowly along road s-axis. See image:

And when lane width is zero, roadmanager will move the vehicle's lane association to another, closest lane. This cause the route to fail.

The risk for hitting that exact point is basically reduced with larger timesteps.

The solution is to add a check for valid lane width, so that the lane change will not happen until the width is over some threshold value.

This should be implemented by commit ccf0b0d. It's merged on master and will be included in next release.

Please test and report if it works better for your use case. Let us know if you need binaries in which case I can provide for testing otherwise you can pull from master and build it yourself.

[MichaelHarth]

Thank you for your answer. In the scenario, I am currently testing, the target lane is wider than 0.5 m, but I still have the issue. I have attached the files so that you can have a look at them.

Can you additionally provide the binaries for testing?

generic_city.zip

[eknabevcc]

Here's is the latest build, based on commit ba3c7d2: esmini.exe (Windows)

I can't reproduce the issue (at least see anything obviously wrong), neither with old nor with new esmini. This is how it looks in replayer:

Commands used:
./bin/esmini --headless --osc generic_city.xosc --fixed_timestep 0.001 --record sim.dat
./bin/replayer --window 60 60 800 400 --res_path . --file sim.dat

What is the issue you encounter?

Anyway, please test with the provided executable and see if it makes any difference.

[MichaelHarth]

The provided executable works for me. However, sometimes it hangs up randomly after some seconds when I activate the visualization. I also had this problem with the previous esmini version.

The issue with the esminiLib is still there. I could figure out the code line where the crash happens. It is line 6902 in the RoadManager, when calling the push_back(road->GetId()) shortly after the FollowRouteController starts with the lane change.

I guess that this could be an issue with the C++ compiler. My code is compiled using C++ 17, which could be more restrictive here than C++ 14. Until now, I could not find a solution how to make the code run. Are there any plans to build esmini with C++ 17 in the future?

As soon as I replace the esminiLib.dll with the one from your pre-built package, the esminiLib works fine for me.

Could you try to reproduce my issue with the esminiLib using a C++ 17 compiler?

[eknabevcc]

Ok. Thanks for testing the exe. I don't think the compiler version should make any difference. It's probably a bug that we should try to find.

First, the graphics freeze is probably unrelated. So for now, let's make a note but leave it out.

The provided exe was a "latest and greatest" that turned out to have some issues. To rule out these changes affects your use case, can you try again with the following binaries, which should be a bit more stable (3f5a67c):
esmini.exe
esminiLib.dll

If issue still exist, I will try reproduce with a simple app built with C++17 (Visual Studio).

[eknabevcc]

Now there is also a release: v2.26.8 that you can test instead of the above provided binaries.

[MichaelHarth]

I could finally find the reason for the crash when calling the push_back in RoadManager.cpp and I also have a solution for it.

The issue only occurs for a specific platform toolset (Visual Studio 2019 V142) in debug mode, while the release build is working. I could find exactly this issue in multiple requests in the Visual Studio forum, e.g. here.

There seem to be several conditions for the problem, which could be the reason why it has never been found before:

• The vector, where the push_back is called (here overlapping_roads), is a member variable of a class (here Position)
• The class is imported from a .dll file (here esminiLib.dll)
• Visual Studio 2019 V142 is used in debug mode

The issue then occurs when calling the push_back multiple times. There seems to be an issue with memory reallocation. At the moment, there does not seem to be a fix for the core issue.

The workaround is to add a local vector overlapping_roads_temp, which is initialized in the beginning, e.g. line 6568, to the function Position::XYZH2TrackPos. Then push_back the road ids to this local vector and finally move the vector into overlapping_roads (overlapping_roads = std::move(overlapping_roads_temp)), e.g. in line 6920.

I am sure that you never have faced this issue because you are using Visual Studio 2019 V141 toolset instead of V142, as the issue only appears in V142. Could you try to reproduce my issue and implement a fix to it? The workaround I have described is working fine for me.

[eknabevcc]

Thanks for the thorough investigation, it's appreciated.

However strangely I can't reproduce. I'm on revision f9b0c1d and this is what I tried:

cmake .. -G "Visual Studio 16 2019" -T v142
cmake --build . --target install --config Debug

Then I run:
./bin/esmini --window 60 60 800 400 --osc generic_city.xosc --fixed_timestep 0.05

Works fine.

I also tried with the following "app" linking esminiLib.dll, all in debug mode. But it also works. Even when changing timestep 0.001.

#include <stdio.h>
#include "esminiLib.hpp"

int main(int argc, char* argv[])
{
    if (SE_Init("/users/eknabe1/Downloads/generic_city/generic_city.xosc", 0, 1, 0, 0) != 0)
    {
        printf("Failed to load scenario\n");
        return -1;
    }

    while(SE_GetSimulationTime() < 20.0)
    {
        SE_StepDT(0.1f);
    }

    SE_Close();

    return 0;
}

Could it be that recent changes, actually also slightly involving the overlap vector (strange coincidence) (b47967e), has affected the reproducibility?

Can you do one final test, also with same application code. Do you still have the issue?

Or is it that I do something "wrong"/different than you?

[MichaelHarth]

You are right that the changes in commit b47967e mask the issue in combination with my provided example files. However, the error still comes up, if you cross multiple junctions and "collect" overlapping roads first. If you then make a lane change and call the push_back five times, the crash still happens. I have attached some files for reproducing the issue when you are using the changes from b47967e.

multiple_intersections.zip

[eknabevcc]

I'm sorry to say that I can't reproduce the issue. The provided scenario will put the car at a road not part of the route, which causes an error that puts the car to a stop.

I did some small code changes to increase robustness slightly (commit 63057d6). At least so that the vehicle will continue to drive, and detect if and when the route is reached.

But even after these changes I don't get a crash.

I suspect that you have a custom code application that influence the behavior of esmini.

I'm thinking that perhaps we could introduce the code fix anyway. But we would really like to understand and see what it fixes first.

So a final attempt:
Can you pull once again from master and build esmini. Then run the scenario, via esminiLib, from a simple while loop (preferably exact the code above) - and see if you still get the problem?

BTW, my VS version is Microsoft Visual Studio Professional 2019, Version 16.11.14

What is yours?

[MichaelHarth]

Sorry for the long delay. I have tried the latest update from the master now and built it. For me, the scenario still works until the lane change is necessary.

I am using esmini within a custom application, but it does not influence the esmini behavior. The main behavior is like the simple while loop in your previous comment (#335 (comment)). I have also tried the simple while loop without the surrounding code which processes the esmini simulation state, but the issue still remains.

I would really appreciate if you could integrate the fix anyway.

My VS version is Visual Studio Professional 2019, version 16.11.4. So, maybe an update of my compiler could also help. I will definitely try that.

[eknabevcc]

Ok, let's not spend more time on this. I applied the "patch". Now I just hope it works :)

It's included in release v2.27.1