tarmac-profile crashes on windows side for large Tarmac files

[Firew0rk]

I build the binaries on Windows using visual studio 2017 as mentioned and when i try to get the profile information for large tarmac trace files (>1.5GB), the tools crash during indexing and it refers me towards this line: https://github.com/ARM-software/tarmac-trace-utilities/blob/main/lib/calltree.cpp#L247

I would like to understand if there is a limitation on how big the trace file can be?

[Arnaud-de-Grandmaison-ARM]

I'm sure it's possible to create a tarmac file large enough to exceed what tarmac-trace-utilities can cope with, but 1.5GB does not seem that big to me, and I believe I've successfully indexed files larger than this, but this was on Linux or MacOS platforms --- I have not tried Windows. Have you been able to index smaller tarmac traces ? Would you have a chance to use a linux build of the tarmac-trace-utilities on the very same 1.5GB tarmac trace to compare ? The problem could come from types like "int" having a different size on Windows vs Linux.

[Firew0rk]

I tried to index smaller files (<500MB) and that seems to work fine. I just tried the larger tarmac file on WSL and i get the same assertion failed error triggered from this line: https://github.com/ARM-software/tarmac-trace-utilities/blob/main/lib/calltree.cpp#L249

Do you want me to try it on a native Linux platform instead of WSL?

[Arnaud-de-Grandmaison-ARM]

Thanks for testing, I'll will try to reproduce on my end --- on a native Linux.

[mickael9]

I think the problem is the use of off_t that is 32-bits on MSVC, and it seems to be used everywhere
It should probably be changed to int64_t

[Arnaud-de-Grandmaison-ARM]

FWIW, I'm running this on MacOS, where off_t happens to be 64-bits --- and everything seems to be working just fine. I'll do a test with a large tarmac trace to confirm it solves the problem. I have a patch almost ready if it just works.

[Arnaud-de-Grandmaison-ARM]

I've pushed a patch (eb710a6) that improves large Tarmac file support. I would be very interested in getting feedback from the Windows / WSL platforms which I have no way to test. For the platforms I've been able to test (MacOS, Ubuntu), tarmac-trace-utilities happily deals with 5.8+G bytes Tarmac trace files.

Switching off_t to int64_t may not always be a solution, because off_t is used both as a file offset and a pointer offset as the index files are memory mapped at some point. This implies that we can only use 64-bit file offsets when the platform actually has 64-bit pointers as well.

[Firew0rk]

Sorry for the delay, i will have a look into it in the upcoming days and get back to you

[Firew0rk]

Hi, i verified the patch and it's not crashing now in WSL. Thanks a lot for your support, perhaps you can merge these changes on the main branch? One another question, when the utility creates the index file, apparently the size of the index file is super large. For example, for ~2GB tarmac trace file, the generated index file is atleast ~12GB. Is this expected or am i missing some options?

[Arnaud-de-Grandmaison-ARM]

Thanks a lot four your feedback, I'll close this issue.

You're not missing any magic option --- index files are large indeed unfortunately.