Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

./configure fails with Unicode build #7229

Closed
dEajL3kA opened this issue Jun 10, 2021 · 19 comments
Closed

./configure fails with Unicode build #7229

dEajL3kA opened this issue Jun 10, 2021 · 19 comments
Labels
build help wanted KNOWN_BUGS material Unicode Unicode, code page, character encoding Windows Windows-specific

Comments

@dEajL3kA
Copy link

dEajL3kA commented Jun 10, 2021

I did this

I am trying to build curl with Unicode support on Windows (using MinGW/MSYS2).

In order to enable Unicode, I added options -munciode and -mconsole to the CFLAGS explicitely, because apparently ./configure doesn't have a dedicated --enable-unicode switch. Or did I miss something?

Unforntunately, this caused ./configure to fail with "No working C compiler" error. My analysis of the config.log showed that ./configure uses a lot of "test" programs that have main() hardcoded, but should be using wmain() when Unicode is used! In the actual curl source codes we already do have the required #ifdefs to use either main() or wmain() as needed. Only the ./configure has problem! My workaround was to replace all occurences of "main" in ./configure with "wmain", using a simple sed command. This does work. After patching configure, the Unicode version of curl compiled without any problem!

sed -i -E 's/\bmain[[:space:]]*\(([^\(\)]*)\)/wmain(\1)/g' configure

But this workaround is rather cumbersome and other people may run into the same issue. So I think it would be much preferable to have an --enable-unicode switch that not only addes -munciode and -mconsole to the CFLAGS automatically but also uses the "correct" main function in the "test" programs. At least make ./configure work with -munciode -mconsole options.

I expected the following

CFLAGS="-municode -mconsole" ./configure && make should "just work" without the need to patch configure.

Even better, ./configure --enable-unicode && make should be available.

curl/libcurl version

7.77.0

operating system

Windows 10, using latest MinGW/MSYS2 compiler + build environment

MINGW32_NT-10.0-19042 3.2.0-340.x86_64 2021-05-24 19:32 UTC x86_64 Msys
gcc version 10.3.0 (Rev2, Built by MSYS2 project)
@dEajL3kA
Copy link
Author

dEajL3kA commented Jun 10, 2021

EDIT: Better solution exists. Please see here!


Possible workaround, if we don't want to patch the configure script, would be using a tiny "shim" library that implements the wmain() function, in order to fix the linker error about missing wmain (or wWinMain) entry point, and that does nothing but invoking the "real" main() function. Libary needs to be added to LIBS environment variable when invoking ./configure.

libwmain.c:

extern int main(int argc, char *argv[]);
int wmain(void) {
	char *args[] = { "a.exe", (void*)0 };
	return main(1, (char**)args);
}

Makefile:

libwmain.a: libwmain.o
	$(AR) rcs libwmain.a libwmain.o
libwmain.o: libwmain.c
	$(CC) -march=i386 -Os -o libwmain.o -c libwmain.c

To build curl:
CFLAGS="-munciode -mconsole" LIBS="-lwmain" ./configure

Note: In my test, this did not collide with curl's own wmain() method, but did fix the conftests.

@jay
Copy link
Member

jay commented Jun 11, 2021

Can you use -DUNICODE -D_UNICODE instead

@dEajL3kA
Copy link
Author

dEajL3kA commented Jun 11, 2021

Can you use -DUNICODE -D_UNICODE instead

Sorry, but nope. Or I am missing something? 😕

  • According to my tests, -municode -mconsole already implies -DUNICODE -D_UNICODE, so explicitely adding -DUNICODE -D_UNICODE to CFLAGS in addition to -municode -mconsole does not change anything for me.

  • Setting only -DUNICODE -D_UNICODE in CFLAGS but not -municode -mconsole causes MinGW/GCC to still link against main() rather than wmain() – this I have just verified! As a result, ./configure does succeed without additional hacks, yes. But, in the end, the linking of curl.exe fails with an undefined reference error – which is expected, because with -D_UNICODE we are telling tool_main.c to generate wmain() instead of main(), but at the same time MinGW/GCC is expecting a main() function due to the lack of -municode -mconsole switches. Hence, setting -DUNICODE -D_UNICODE alone is not sufficient 😞

This was tested with latest MinGW/GCC:
gcc version 10.3.0 (Rev2, Built by MSYS2 project)

Best regards.

@jay
Copy link
Member

jay commented Jun 13, 2021

It seems this is the way curl-for-win does it:

  CURL_CFLAG_EXTRAS="${CURL_CFLAG_EXTRAS} -DUNICODE -D_UNICODE"
  CURL_LDFLAG_EXTRAS_EXE="${CURL_LDFLAG_EXTRAS_EXE} -municode"

So we'd have to apply -municode only for the curl tool?

/cc @vszakats

@dEajL3kA
Copy link
Author

dEajL3kA commented Jun 14, 2021

So we'd have to apply -municode only for the curl tool?

Yes, I think so. But it seems that CURL_LDFLAG_EXTRAS_EXE does nothing for me. I did a full-text search on both, configure and the resulting Makefile, but could not find the identifier CURL_LDFLAG_EXTRAS_EXE at all. Does it even exist in "vanilla" curl?

However, what does work for me is:

CFLAGS="-DUNICODE -D_UNICODE" ./configure
make curl_LDFLAGS="-all-static -municode -mconsole"

So that probably is the way to go. Still, figuring this out was wearisome. Maybe add a ./configure option for Unicode build?

Thank you and best regards.

(BTW: What is the difference between CURL_CFLAG_EXTRAS and CFLAGS variables?)

@jay
Copy link
Member

jay commented Jun 14, 2021

Maybe add a ./configure option for Unicode build?

Seems like a good idea.

(BTW: What is the difference between CURL_CFLAG_EXTRAS and CFLAGS variables?)

I think it was added to pass cflags without overriding the default CFLAGS. It's not documented and I may be wrong...

@vszakats
Copy link
Member

vszakats commented Jun 14, 2021

CURL_LDFLAG_EXTRAS_EXE is Makefile.m32 specific, so it indeed doesn't do anything with ./configure builds.

CURL_CFLAG_EXTRAS is also something used only in Makefile.m32. This build method doesn't support a CFLAGS from the environment, so this "extra" envvar is used to inject extra C flags into it.

@bagder
Copy link
Member

bagder commented Oct 2, 2021

This issue seems to have stalled. Is there anything we can/should do about this report? Any proposed next steps?

@vszakats
Copy link
Member

vszakats commented Oct 3, 2021

I think it'd be best to complete Unicode support in libcurl and then fix the different build systems to enable it correctly. Though these two can be done in any order or in parallel. @jay had a nice patch for the former, though I continue to think that fall-back methods are bad and should not be used in libcurl. (Yes, OpenSSL uses one such method, but OpenSSL also keeps at least one Windows vulnerability open since years without intention to fix it (along some other past woes on Windows), so I think it should not be regarded as the yardstick for Windows-development specifically.)

@dEajL3kA
Copy link
Author

dEajL3kA commented Oct 5, 2021

I think it'd be best to complete Unicode support in libcurl and then fix the different build systems to enable it correctly. Though these two can be done in any order or in parallel. @jay had a nice patch for the former, though I continue to think that fall-back methods are bad and should not be used in libcurl.

👍 ⚡

@curl curl temporarily blocked Thanayaby Nov 20, 2021
@curl curl deleted a comment from Thanayaby Nov 20, 2021
@bagder
Copy link
Member

bagder commented Sep 24, 2022

Anyone care to write this issue up in a brief summary for adding to the KNOWN_BUGS document?

@jay
Copy link
Member

jay commented Sep 25, 2022

IMO this isn't a bug, it's either an enhancement or a feature request. The user wants an easier way to tell autotools to build curl with Windows Unicode support, like ./configure --enable-windows-unicode

@vszakats vszakats added the Unicode Unicode, code page, character encoding label Feb 16, 2023
@vszakats
Copy link
Member

vszakats commented Mar 11, 2023

I tried to fix this without the sed trick, and tested: -Dmain=wmain.
Oddly enough, it worked!

CPPFLAGS="${CPPFLAGS} -Dmain=wmain"
CPPFLAGS="${CPPFLAGS} -DUNICODE -D_UNICODE"
LDFLAGS="${LDFLAGS} -municode"

It'd be better to restrict -Dmain=wmain internal to the ./configure run, if possible.

If this is a feasible solution, adding an --enable-windows-unicode option is just the icing on the cake.


This new warning came up, unrelated to either of the above hacks:

../../src/tool_main.c:239:5: warning: no previous prototype for function 'wmain' [-Wmissing-prototypes]
int wmain(int argc, wchar_t *argv[])
    ^
1 warning generated.

vszakats added a commit to vszakats/curl that referenced this issue Mar 11, 2023
llvm/clang and gcc doesn't recognize the wmain() function in Unicode
Windows builds:

```
../../src/tool_main.c:239:5: warning: no previous prototype for function 'wmain' [-Wmissing-prototypes]
int wmain(int argc, wchar_t *argv[])
    ^
1 warning generated.
```

Before this patch, we already silenced it with CMake. This patch moves
the silencing to the source, so that it applies to all build tools.

Bug: curl#7229 (comment)

Closes #xxxxx
vszakats added a commit that referenced this issue Mar 11, 2023
llvm/clang and gcc doesn't recognize the wmain() function in Unicode
Windows builds:

llvm/clang:
```
../../src/tool_main.c:239:5: warning: no previous prototype for function 'wmain' [-Wmissing-prototypes]
int wmain(int argc, wchar_t *argv[])
    ^
1 warning generated.
```

gcc:
```
../../src/tool_main.c:239:5: warning: no previous prototype for 'wmain' [-Wmissing-prototypes]
  239 | int wmain(int argc, wchar_t *argv[])
      |     ^~~~~
```

Before this patch, we already silenced it with CMake. This patch moves
the silencing to the source, so that it applies to all build tools.

Bug: #7229 (comment)

Reviewed-by: Marcel Raad
Closes #10744
vszakats added a commit to curl/curl-for-win that referenced this issue Mar 11, 2023
@bagder bagder closed this as completed in 7128ae8 May 22, 2023
bch pushed a commit to bch/curl that referenced this issue Jul 19, 2023
llvm/clang and gcc doesn't recognize the wmain() function in Unicode
Windows builds:

llvm/clang:
```
../../src/tool_main.c:239:5: warning: no previous prototype for function 'wmain' [-Wmissing-prototypes]
int wmain(int argc, wchar_t *argv[])
    ^
1 warning generated.
```

gcc:
```
../../src/tool_main.c:239:5: warning: no previous prototype for 'wmain' [-Wmissing-prototypes]
  239 | int wmain(int argc, wchar_t *argv[])
      |     ^~~~~
```

Before this patch, we already silenced it with CMake. This patch moves
the silencing to the source, so that it applies to all build tools.

Bug: curl#7229 (comment)

Reviewed-by: Marcel Raad
Closes curl#10744
bch pushed a commit to bch/curl that referenced this issue Jul 19, 2023
vszakats added a commit to vszakats/curl that referenced this issue Aug 9, 2024
vszakats added a commit that referenced this issue Aug 10, 2024
- add `--with-windows-unicode` and `--without-windows-unicode` (default)
  options.

- enable it in a CI job.

Fixes #7229
Closes #14478
@lordmulder
Copy link
Contributor

lordmulder commented Sep 12, 2024

Just tried the new --enable-windows-unicode configure option in 8.10.0.

Still same old problem: Configure script doesn't find "main" function, because, in Windows, when Unicode is enabled, the main function must be wmain() rather main(), but it's apparently hard-coded to main() in your configure script:

C:/M/B/src/mingw-w64/mingw-w64-crt/crt/crtexewin.c:67:(.text+0xc1): undefined reference to `wWinMain@16'
collect2.exe: error: ld returned 1 exit status
configure:5018: $? = 1
configure:5058: result: no
configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME "curl"
| #define PACKAGE_TARNAME "curl"
| #define PACKAGE_VERSION "-"
| #define PACKAGE_STRING "curl -"
| #define PACKAGE_BUGREPORT "a suitable curl mailing list: https://curl.se/mail/"
| #define PACKAGE_URL ""
| /* end confdefs.h.  */
| 
| int main (void)
| {
| 
|  ;
|  return 0;
| }
configure:5063: error: in `/d/_Sandbox/curl/build/x86/curl':
configure:5065: error: C compiler cannot create executables
See `config.log' for more details

Hence, the following hack still is required:

sed -i -E 's/\bmain[[:space:]]*\(([^\(\)]*)\)/wmain(\1)/g' configure

@vszakats
Copy link
Member

vszakats commented Sep 12, 2024

curl has a CI job that builds and tests fine without this hack:
https://github.com/curl/curl/actions/runs/10835445722/job/30067013990#step:8:936

I've found that wmain is only necessary when linking with -municode. The merged implementation does that only for the curl tool, so everything else continues to build with main, as before.

The hack also doesn't address building curl tests, which all use main.

@lordmulder
Copy link
Contributor

lordmulder commented Sep 13, 2024

I've found that wmain is only necessary when linking with -municode

Yeah, that's right. But -municode must be added to LDFLAGS in order to produce a working Unicode binary.

If -municode is not added to LDFLAGS, then the normal main() function will be used, which might avoid the linking issues with the Configure script, but it also means that the command-line will be mangled (i.e. any non-US-ASCII character will be replaced by a ? character!) even before it arrives in "main" function. So, even if all the rest is compiled as Unicode, we have a problem!

To actually get the untainted command-line into the "main" function, as Unicode (UTF-16), we have to use a wmain() function, which requires -municode to be added to LDFLAGS (with MinGW-w64) – which then breaks the Configure script, since all the C code generated/executed by the Configure script apparently expects a main() function in a "hardcoded" way.

Therefore, I think that --enable-windows-unicode shall be adding -municode to the LDFLAGS. But then we also need something in the Configure script that conditionally (i.e. when --enable-windows-unicode is used and thus -municode is in effect) generates wmain() functions instead of main(). Until that is implemented, manually patching Configure is still required...

(I posted the result when -municode is used, but Configure it is not manually patched above. As you can see, it's failing right away. Currently, passing the new option --enable-windows-unicode apparently does nothing to fix this issue)


I have a workaround that works for me. But I think a proper solution could be great for other users that face the same issue.

@vszakats
Copy link
Member

vszakats commented Sep 13, 2024

I've found that wmain is only necessary when linking with -municode

Yeah, that's right. But -municode must be added to LDFLAGS in order to produce a working Unicode binary.

As mentioned in the previous post, -municode is added for the curl tool automatically. The curl tool does use wmain():

curl/src/Makefile.am

Lines 54 to 56 in 8ca6030

if USE_UNICODE
AM_LDFLAGS += -municode
endif

curl/src/tool_main.c

Lines 218 to 228 in 8ca6030

#ifdef _UNICODE
#if defined(__GNUC__)
/* GCC does not know about wmain() */
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wmissing-prototypes"
#pragma GCC diagnostic ignored "-Wmissing-declarations"
#endif
int wmain(int argc, wchar_t *argv[])
#else
int main(int argc, char *argv[])
#endif

[ Generally speaking: -municode controls the app startup code, and Unicode app code can live without it if it doesn't want to get a wide argv via main. There are other means to retrieve the command-line in Unicode too. So no, -municode isn't required for a Unicode app. The curl tool uses it, the feature tests and regression tests don't. ]

(I posted the result when -municode is used, but Configure it is not manually patched above. As you can see, it's failing right away. Currently, passing the new option --enable-windows-unicode apparently does nothing to fix this issue)

You seem to be passing LDFLAGS=-municode to the build and that is what breaks it for you. The solution is to drop this setting and use --enable-windows-unicode only.

Have you tried such build?

@lordmulder
Copy link
Contributor

lordmulder commented Sep 13, 2024

Ah, I see now. You mean we can leave out -municode from the LDFLAGS for the ./configure invocation now (something that was absolutely required in the past!) and then --enable-windows-unicode will sneak it in again, but only for linking the final cURL executable? Will have to give this a try, and see if there are any undesired side effects with any of the dependencies...

@lordmulder
Copy link
Contributor

Ah, I see now. You mean we can leave out -municode from the LDFLAGS for the ./configure invocation now (something that was absolutely required in the past!) and then --enable-windows-unicode will sneak it in again, but only for linking the final cURL executable? Will have to give this a try, and see if there are any undesired side effects with any of the dependencies...

Indeed, seems to work 🥳

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
build help wanted KNOWN_BUGS material Unicode Unicode, code page, character encoding Windows Windows-specific
Development

Successfully merging a pull request may close this issue.

5 participants