Make it build #1
Comments
|
The first problem: qtwebengine requires at least GCC 5 while we still have GCC 4.9. So far, I disabled this check and we will see where it gets us. Porting a fresh version of GCC is a todo since long so we might want to finally do that. |
|
And the second separate task is porting the ninja build tool used by the Chromium source code. It definitely requires some aligning but nothing fancy. Will create a separate ticket for it. |
|
All necessary tools are ported and now I'm fully concentrated on building Chromium itself. I have some work in progress which I will be committing in pieces as I go and report to this ticket (creating separate tickets as needed). |
|
I'm getting fork failures when running Python from GN: This is kind of weird. 0xfffffffc is -4 which might mean be EINTR. Why — not clear so far. |
|
Tests show that GN indeed calls I think that the best way to overcome it right now is use LIBCx |
LIBC fork is not thread safe and GN starts a lot of threads. Use spawn2 from LIBCx instead. Needed for bitwiseworks/qtwebengine-os2#1. Closes #1.
LFLAGS may contain quotes on some platforms (e.g. on OS/2 in switches like -Zlinker "DISABLE 1121"). Needed for #1.
|
All build toolchain (GN/Qmake) issues are solved. It's time to fix various build breaks here and there now. |
|
I'm getting problems with linking libraries because GN apparently creates response files with all contents on a single line. I guess I have to hack GN to make it do it per-line (it should be libiberty compatible and work on all platforms). |
This is necesasry since EMX tools expect arguments one per line (LIBC _response/-Zargs-resp API logic). This commit also removes unsupported options for emxomfar. Needed for bitwiseworks/qtwebengine-os2#1.
|
The above commit fixes link issues, going further. |
|
I discovered one more LIBC flaw, |
|
A new build break in boringssl due to missing socklen_t in LIBC. Created bitwiseworks/libc#62 for that. |
|
Several more helper libraries built. Now I'm facing a preconfigure problem in a third party library again. One could think that running The only way to go here is to take the original source tree (of exactly the same version) and add missing files. It's a boring monkey work. Thanks, Chromium guys. |
|
We decided to roll out I will also create a dedicated ticket for building Chromium (in the respective repository) and this one will cover porting the rest of the Qt wrapper part. |
This is a counterpart of bitwiseworks/qtwebengine-chromium-os2@fbcfb54. Needed for #1 and bitwiseworks/qtwebengine-chromium-os2#3.
This also includes usual DLL name shortening with TARGET_SHORT. Needed for #1 and for bitwiseworks/qtwebengine-chromium-os2#3.
|
Hard to say exactly. Some debugging needed. Weakld is still reading objects from liibraries here, I'm not interrupting it. I will leave it overnight. What I can say now is that this procedure is very slow. It's taking many seconds for some object files. No surprise it's taking that long. It has to be optimized. Impossible to go on like that. Even a second per one object file is way too much. It gives us 15k seconds which is 4 hours... |
|
It didn't end with anything good. Overall it took 9 (!!!) hours to get to the point where I found it dead (hard system hang with cold reboot). The log file is 432 MB and here are its last lines: This looks like some debug messages from WL.EXE (due to I could try ILINK isntead of WL but chances are low that it will succeed. And first I have to optimize EMXOMFLD — no way to wait for many hours upon each try. |
|
This is the list of undefined symbols I get from a semi-successive run of WL (see bitwiseworks/libc#84 (comment)): It's 180 symbols, not that much. Not only Chromium itself, but our C++ library seems to miss something. |
|
Missing libstdc exports is quite strange as I see them in the original |
|
It turned to be the old C++ library from some test build of GCC which got on the way. Removing a path to it from EMXOMFLD options made the problem go away: this means, that EMXOMFLD does not convert the library now and simply uses the |
|
Ok, with the right library, all missing |
|
Status update: I've fixed more than a half of missing symbols (mostly by adapting Fuchsia stubs to OS/2 to speed it up). However, I have to recompile hundreds and hundreds of source files (as headers change in many cases) and I'm constantly experiencing OOM issues from both EMXOMF and WL so it's going really slow. |
|
I'm trying the release build now and it's going much faster. Approximately, 2500 compilation units in 30 min. Which gives a total duration of 3 hours per 15k of them. Not bad, 2-3 times faster than the debug build (may be even more). |
|
I was way too optimistic it seems. The first 10k units were built in roughly 2.5 hours but then I had a build break and performance significantly degraded down to just 500 units in the next 3 hours. I have no idea if it's just the complexity of C++ or something else. Judging by the CPU load, it still loads it almost completely. I will try to break the build and reboot to see if it helps. |
|
Note that the original release build failed with OOM too. And looking at options shows that it uses -g1 (debug info w/o local vars). Our EMXOMF ignores debug level and also does everything hence an OOM on big files (bitwiseworks/libc#82 -- I guess we should add such support, it meets the task mentioned in that ticket and could be a partial solution). I had to hack the release config to explicitly use -s. After that I got a single build run of the whole QtWebEngine module in 14 hours 30 min w/o any OOMs. It failed on linking but that's expected as not all symbols are there yet. But I must say that the fixed WL from bitwiseworks/libc#84 runs much, much faster. Just several minutes. Yes, it's w/o the debug info but still. Good results, I will finish the missing symbols now. |
|
All missing symbols are finished within bitwiseworks/qtwebengine-chromium-os2#3. There are also a few bits on the Qt side to compile (the Qt Web Engine Widgets DLL) but they should be pretty straight forward since we have everything now. |
|
With the above commits, EVERYTHING is successfully built now. Including examples and tests. Starting Somewhat expected. I need to see where this error comes from but the fact is that OS/2 successfully loads all Qt DLLs for this executable including Qt5WebC.dll which contains Chromium and is of 250 MB in size! And I'm not using the high memory bit so far. I will debug that on Monday, doesn't look serious. |
|
Above was a usual lack of the .exe extension. With that fixed, I can run further but getting some problems with IPC communication between the Qt app and the web process (also expected): We need to align it to use socketpair instead of poll (or a proper poll impl) etc. |
|
Also, seems that EMXEXP doesn't like |
|
Also, both EXEHDR from OS/2 Toolkit and WDUMP from OpenWatcom 1.9 fail to show all the exports from |
|
If you provide a copy of the failing Qt5WebC.dll and tell me which exports
wdump appears to be missing, I can take a look.
FWIW, this may or may not be related to a recent message from Elbert Pol
on ecs-technical with the subject:
[eCS-Technical] Error
The gist of Elbert's message was that his QT app build blew up with:
Error! E2082: file
D:\Test\rssguard\os2\src\librssguard\rcc\qrc_sql.obj(D:\Test\rssguard\os2\src\librssguard\rcc\qrc_sql.cpp),
record 24: invalid record type 0x0049
A bit of poking about revealed that emxomf generated a broken obj file
because emxomf did not split a too large LE record properly when writing
qrc_sql.obj.
What I told him in my reply to his message was:
<SNIP>
Using dmpobj, we see that record 24 ends correctly with:
000003e0 69|64 29|0a 57|48 45|52 45|20 46|65 65|64 73|2e <id).WHERE
Feeds.>
000003f0 63|75 73|74 6f|6d 5f|69 64 <custom_id>
However record 25: starts 0x49 and is clearly missing the requied record
header.
**??**386(49) recnum:25, offset:000006cfh, len:2053h, chksum:5fh(a1)
00000000 4e|55 4c|4c 20|4f 52|20 46|65 65|64 73|2e 63|75 <NULL OR
Feeds.cu>
00000010 73|74 6f|6d 5f|69 64|20 3d|20 27|27 3b|0a 2d|2d <stom_id =
'';.-->
</SNIP>
The command line indicated he was building with -Zomf, so I am assuming
emxomf is at fault here. Emxomf appears to have detected the need to
split, but neglected to insert the LE header before outputting the tail of
the split recoard.
I asked Elbert to submit a ticket, but I've not seen one yet, so I figured
it's time to tell you now.
|
QtWebC (Qt5WebC[d]) is used by QtWebEngineCore. Needed for bitwiseworks/qtwebengine-os2#1.
|
Turns out that the DLL is loaded (and resolved by OS/2) fine. The cause for error 127 in POPUPLOG is different: there was a DLL name clash with QtWebChannel (the one that actually exports Steven, thanks for the help! I've uploaded the Qt5WebC.DLL for you for checking as http://rpm.netlabs.org/test/Qt5WebC.zip. Note that WDUMP seems to work fine now (it's just 648 exports in that DLL and this is normal -- all Chromium code is "private" there, Qt only exports its own APIs). EXEHDR and EMXEXP still fail. But they don't prevent me from going further at least: EMXEXP is not used in Qt. We use EMXIMP to generate import libraries and EMXIMP works fine with this huge DLL. I'm pretty sure EMXEXP is easily fixable though. But anyway, it's interesting what your checking of this DLL will tell us in terms of possible future problems. |
|
I see the same 648 exports with both wdump -x and lxlite -c:exemap -f-.
Oddly exehdr does not die here and shows 684 exports. This exehdr reports
itself as:
Operating System/2 Executable File Header Utility
Version 4.01.003 Dec 11 2003
Copyright (C) IBM Corporation 1988-2003
Copyright (C) Microsoft Corp. 1988-1992.
All rights reserved.
1-05-04 2:54 82,461 0 exehdr.exe
44f5904ef91a9e1597920054cf9accd9 *exehdr.exe
I don't expect any problems with wdump.exe based on file size since it
does not need to keep much of the binary file in memory.
That said, wdump -a dies with
page # 520 map page = 44C60C46H size = 0C24H flgs = 00H Unknown
segment # 2520 offset: 4628EE36
===========
Error! Couldn't read from executable: reached EOF.
I should have time to look at this tomorrow and figure out where the blame
lies.
You might notice the discrepancy between page # and segment # in the
above. It turns out the developers decided to limit page # to 3 digits.
I probably seemed like a reasonable limit in the '80s. This has been
fixed.
BTW, do you have any uncommited changes to emxomf or emxomfld? I am
thinking of your debug data hack that breaks some object files. Perhaps,
I can find time to fix this properly.
Regarding wlink speed, I am still designing the symbol table rework. The
current plan is count the number of objects and library files and use this
to size the hash table at startup. In addition, I am going rework the
hash collison handling to use a resizable array of symbol table entries
rather than a linked list. Since we have well over 1 million symbols,
this will save more than 4MB of RAM.
I'm also considering optionally Huffman coding the symbols or something
similar using a predefined coding table. This will significantly reduce
the size of the symbol table at the cost of some processing time
|
|
Re EXEHDR, I wonder where you got this. I have: (from the OS/2 4.5 TK coming with eCS 2.0). Re uncommitted changes. In EMXOMF there is an uncommitted hack which breaks linking of some OBJ and I'm not going to commit it. In EMXOMFLD - yes, I have fixes for hash tables but they need some brushup. Will do it today. I was also thinking about counting the symbols in EMXOMFLD/WEAKLD and making the hash table size dependent on it but then I decided it's not worth it. If there is a large number of symbols, we will need the largest hash table anyway. And if there is not, then allocating even 4MB will not hurt at all (as there will be plenty of free memory and no need for it elsewhere). So I'm going to make it constant (with a 49999 prime, see bitwiseworks/libc#83). What I am going to do though is make this allocation dynamic. EMXOMFLD starts WL after it's done weak symbol processing and that guy will need as much as possible for huge things like Chromium so every MB counts. My plan is to free those tables before launching the actual linker executable. |
|
As everything builds now, it's time to close this issue. Runtime issues are to be reported via separate tickets. |
|
I had no recollection of where this version of exehdr came from, so I had
to do some hunting. It appears to be a build provided by Scott Garfinkle
when he add /highmem support. I uploaded a copy to
http://www.warpcave.com/exehdr_exe-20040105.zip
if you find it useful.
I start to worry when I don't recall exactly what happened 16 years ago.
|
|
Thanks for the link, Steve! |
A generic task for building this module.
Needed for bitwiseworks/qt5-os2#10.
The text was updated successfully, but these errors were encountered: