PyCommandlet.cpp compile error with wcsncpy

[slartibaartfast]

While using UnrealEngine preview branch, I'm seeing the below error after adding the plugin to a project and compiling.
This is on a linux machine, ubuntu 18.04. The wchar.h file is part of the new toolchain.

[33/220] Compile PyCommandlet.cpp
[34/220] Compile PyPawn.cpp
[35/220] Compile PyUserWidget.cpp
/home/trota/Documents/UnrealProjects/test421a/Plugins/UnrealEnginePython/Source/UnrealEnginePython/Private/PyCommandlet.cpp:94:3: error: no matching function for call to 'wcsncpy'
wcsncpy(argv[i], *PyArgv[i].ReplaceEscapedCharWithChar(), PyArgv[i].Len() + 1);
^~~~~~~
/home/trota/Code/UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v12_clang-6.0.1-centos7/x86_64-unknown-linux-gnu/usr/include/wchar.h:150:17: note: candidate function not viable: no known conversion from 'const TCHAR *' (aka 'const char16_t *') to 'const wchar_t *__restrict' for 2nd argument
extern wchar_t *wcsncpy (wchar_t *__restrict __dest,
^
1 warning generated.

I tried changing line 94 of PyCommandlet.cpp in various ways, but either received a similar error or caused python to crash. None of these worked...

    //wcsncpy(argv[i], UTF8_TO_TCHAR(*PyArgv[i].ReplaceEscapedCharWithChar()), PyArgv[i].Len() + 1);

    //wcsncpy(argv[i], TCHAR_TO_UTF8(*PyArgv[i].ReplaceEscapedCharWithChar()), PyArgv[i].Len() + 1);

    //wcsncpy(argv[i], (const wchar_t *)*PyArgv[i].ReplaceEscapedCharWithChar(), PyArgv[i].Len() + 1);


    //wcsncpy(argv[i], (wchar_t *)*PyArgv[i].ReplaceEscapedCharWithChar(), PyArgv[i].Len() + 1);

    //wcsncpy(argv[i], (wchar_t*)*PyArgv[i].ReplaceEscapedCharWithChar(), PyArgv[i].Len() + 1);

Do I need to add a definition for wcsncpy? Or something else? Thanks in advance!

[ishyesh]

wsncpy expects wchar_t in the 2nd position, setting that line to
wcsncpy(argv[i], (const wchar_t *) *PyArgv[i].ReplaceEscapedCharWithChar(), PyArgv[i].Len() + 1);
(as you tried also) worked for me.

[slartibaartfast]

Thanks @ishyesh I will try adding (const wchar_t *) again.

Which toolchain are you using?

[ishyesh]

@slartibaartfast I had to make two changes:

1. In PyCommandlet.cpp, I changed the offending line to
   wcsncpy(argv[i], (const wchar_t *) TCHAR_TO_ANSI(*PyArgv[i].ReplaceEscapedCharWithChar()) , PyArgv[i].Len() + 1);

2. In UnrealEnginePython.cpp, I changed the line above PySys_SetArgV (around ~120 ish) to:
   argv[i] = (wchar_t *)(TCHAR_TO_ANSI(*Args[i]));

I'm a complete newb when it comes to string formats, so take the following as a hypothesis, but the following is what I suspect:

1. The TCHAR array is indexed in the offending lines, which returns a char (?)
2. The char is then cast to wchar_t, which I suspect is retaining the original encoding of TCHAR. Hence, applying fix Is this limited to scripting game logic at the moment? #1 allows compilation to happen, but does not change the underlying binary encoding to be ANSI, as PySys_SetArgV expects (?)
3. So, applying change Is this limited to scripting game logic at the moment? #1 compiles, but starting the Editor with the plugin enabled makes PySys_SetArgV complains about characters not being in the unicode set
4. Hence, I realized that maybe the encoding TCHAR needs to be converted to something standardized. So, I put a TCHAR_TO_ANSI conversion and now UE4 loads and the plugin works!

TL, DR:
TCHAR -> ANSI encoding -> wchar_t fixes this.

My toolchain:

1. UnrealEngine/Engine/Extras/ThirdPartyNotUE/SDKs/HostLinux/Linux_x64/v12_clang-6.0.1-centos7 is utilized
2. I'm on Ubuntu 18.04, 4.18.13
3. Python 3.7 was linked (i.e. libpython3.7m)

[slartibaartfast]

Hey sounds like you got it sorted! Thanks for thinking it through.
I'll try the same edits now.

[slartibaartfast]

After implementing the changes you @ishyesh suggested to PyCommandlet.cpp and UnrealEnginePython.cpp UnrealEnginePython compiled without error. Thanks!

I must be doing something wrong though, as the editor crashes when loading the plugin with this error...

Fatal Python error: no mem for sys.argv
ValueError: character U+6d6f682f is not in range [U+0000; U+10ffff]

Current thread 0x00007f018a3e0440 (most recent call first):
Signal 6 caught.
Malloc Size=65538 LargeMemoryPoolOffset=65554
CommonUnixCrashHandler: Signal=6
Malloc Size=65535 LargeMemoryPoolOffset=131119
Malloc Size=68112 LargeMemoryPoolOffset=199248
Malloc Size=530512 LargeMemoryPoolOffset=729776
[2018.11.28-21.04.55:754][ 0]LogCore: === Critical error: ===
Unhandled Exception: SIGABRT: abort() called

[2018.11.28-21.04.55:754][ 0]LogCore: 0x00007f017d93ce97 libc.so.6!gsignal(+0xc6)
0x00007f017d93e801 libc.so.6!abort(+0x140)
0x00007f00fff7d8e4 libpython3.6m.so.1.0!UnknownFunction(0x698e3)
0x00007f010005b6bf libpython3.6m.so.1.0!PySys_SetArgvEx(+0x12e)
0x00007f01009eed28 libUE4Editor-UnrealEnginePython.so!FUnrealEnginePythonModule::UESetupPythonInterpreter(bool) [/home/trota/Documents/UnrealProjects/test421release/Plugins/UnrealEnginePython/Source/UnrealEnginePython/Private/UnrealEnginePython.cpp:125]
0x00007f01009f18da libUE4Editor-UnrealEnginePython.so!FUnrealEnginePythonModule::StartupModule() [/home/trota/Documents/UnrealProjects/test421release/Plugins/UnrealEnginePython/Source/UnrealEnginePython/Private/UnrealEnginePython.cpp:466]
0x00007f0189515962 libUE4Editor-Core.so!FModuleManager::LoadModuleWithFailureReason(FName, EModuleLoadResult&) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Core/Private/Modules/ModuleManager.cpp:526]
0x00007f018382369d libUE4Editor-Projects.so!FModuleDescriptor::LoadModulesForPhase(ELoadingPhase::Type, TArray<FModuleDescriptor, FDefaultAllocator> const&, TMap<FName, EModuleLoadResult, FDefaultSetAllocator, TDefaultMapHashableKeyFuncs<FName, EModuleLoadResult, false> >&) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Projects/Private/ModuleDescriptor.cpp:592]
0x00007f0183832e4f libUE4Editor-Projects.so!TryLoadModulesForPlugin(FPlugin const&, ELoadingPhase::Type) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Projects/Private/PluginManager.cpp:920]
0x00007f01838329d7 libUE4Editor-Projects.so!FPluginManager::LoadModulesForEnabledPlugins(ELoadingPhase::Type) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Projects/Private/PluginManager.cpp:987]
0x00000000002477dc UE4Editor!FEngineLoop::LoadStartupModules() [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:2897]
0x000000000023024c UE4Editor!FEngineLoop::PreInit(char16_t const*) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Launch/Private/LaunchEngineLoop.cpp:2310]
0x0000000000240848 UE4Editor!GuardedMain(char16_t const*) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Launch/Private/Launch.cpp:135]
0x00007f017def9eea libUE4Editor-UnixCommonStartup.so!CommonUnixMain(int, char**, int ()(char16_t const)) [/home/trota/Code/UnrealEngine/Engine/Source/Runtime/Unix/UnixCommonStartup/Private/UnixCommonStartup.cpp:245]
0x00007f017d91fb97 libc.so.6!__libc_start_main(+0xe6)
0x000000000022c029 UE4Editor!_start()

Testing on a Windows 10 machine, there were no errors - the plugin compiled and loaded.

Do you think I should be using a different installation of python, or maybe try Unicode instead of ANSI? I'm not sure what it expects either.

Edit: Using TCHAR_TO_UTF8 in place of TCHAR_TO_ANSI resulted in the same error.

[ishyesh]

@slartibaartfast After cleaning everything up and re-making, I got the same errors as you did (it compiles, but UE4 can't launch)

This time around, I found out there also exists a TCHAR_TO_WCHAR macro. I replaced the TCHAR_TO_ANSI to TCHAR_TO_WCHAR, and things seem to work as expected now. Of course, with such a vast toolchain and body of code, this may just be a fluke that it worked.

[slartibaartfast]

@ishyesh Your persistence paid off! Using TCHAR_TO_WCHAR let the plugin compile and load.

I also tried the change on a windows 10 machine without error. I'm not able to test on a mac right now.

Would you be able to submit a PR with these changes?

[dfb]

Just a heads up that this change breaks UE4.19 (and earlier) compatibility because TCHAR_TO_WCHAR wasn't added until UE4.20.

[slartibaartfast]

I'll submit a pr tomorrow that is version aware. Thanks for pointing this out.