*WIP* Make-based build system

[ismaell]

A more traditional way to build it. Please comment.

The flags might be outdated, it has been over a year, but it still seems to build.

Something not included is a step to remove ../temp/ from the #include directives, as it didn't seem appropriate to do it there...

NOTE: Sadly I couldn't test the result this time as it's producing a segmentation fault.

[ismaell]

Anyone tried it?

[ghost]

I just tried it. Is this meant to replace the existing shell/command script because currently the shell script does a lot more:

1. Install any dependencies that are missing using the native package manager, assuming the script detects your platform correctly (Linux only)
2. Build LibQB
3. Build the font stuff requiring FreeType
4. Build user mods
5. Build the included version of FreeGLUT (except on OS X where Apple's native GLUT implementation is used)
6. Build QB64

Your Makefile appears to only do the last step, and it doesn't copy anything from internal/source to internal/temp, resulting in errors like the following with a fresh checkout:

g++ -DDEPENDENCY_LOADFONT -DFREEGLUT_STATIC -iquote /home/kit/Downloads/qb64-git/internal/source   -c -o qbx.o /home/kit/Downloads/qb64-git/internal/c/qbx.cpp
/home/kit/Downloads/qb64-git/internal/c/qbx.cpp:1005:11: fatal error: ../temp/regsf.txt: No such file or directory
  #include "../temp/regsf.txt"
           ^~~~~~~~~~~~~~~~~~~
compilation terminated.
make: *** [<builtin>: qbx.o] Error 1

If I invoke make after using the setup_lnx.sh shell script on Linux, it works, assuming I have FreeGLUT installed (which likely should be a required package anyway; it's probably included only for Windows).

While I understand a Makefile won't do everything listed above, it should at least do steps 2, 4, and 6 in my opinion. I haven't looked at the FreeType stuff in detail, so I'm unsure if that's also a requirement or not.

[FellippeHeitor]

We are probably sticking with the default script for now, based on the feedback above.

@ismaell: if you have anything else to add to the original pull request that we haven't been able to take into consideration yet, please let me know, or I'l bel closing the request.

[ismaell]

@chronokitsune3233 It's meant to ease packaging for the OS.

It could complain about dependencies, but it shouldn't build anything external.

Why are sources copied from internal/source to internal/temp? When I was working on this, I just ran sed -i 's:../temp/::' on the files.

[ismaell]

@FellippeHeitor It has been open for quite some time, wouldn't hurt to keep it open some more. I would like to improve it further.

[ismaell]

I never dug into the internals of QB64...

Should libqb be a shared object? Is internal/source guaranteed to be in sync with qb64.bas or should I use the first build to bootstrap another build?

[mkilgore]

Hey, I saw this and though I'd chime in. I wrote/rewrote the original ./setup_lnx.sh that's in the repo today and I've write a fair amount of Makefiles.

It would definitely make sense to replace part of the bulid scripts with a make based install. Everything between lines 157 and 203 are what I would expect the Makefile build system to do (at least). With that, for Linux at least, adding a make install target that installs qb64 to the file system would be nice and make it play better on Linux (You could just shove QB64 in /opt, though IIRC I remember fighting with QB64 to make that work a long time ago, so YMMV. I believe the various copying and writing to /internal/temp when compiling a program was part of the problem). You could install the qb64.desktop file with make install as well. Once that's done all the script would do is just install any dependencies, download the source (If necessary), and run make and make install.

The fact that we don't grab a package for freeglut is likely an error, probably by me. If you check the package names for the distros we support in ./setup_lnx.sh we just have to grab the 'freeglut-dev' or 'freeglut-devel' or etc. whatever the package name is for that distro. Unfortunately, checking and testing all the distros takes a decent chunk of time, I remember that being the most annoying part of writing that script.

As for making libqb.o a shared object instead, it's not a horrible idea, but it would mean that QB64 executables are not as easily redistributable. Keep in mind that unlike Windows, Linux will not look for .so files in the same directory as the executable unless you explicitly tell it to at run time (By using the LD_LIBRARY_PATH env variable). This is a bit annoying, and makes it hard to make easily portable executables that include libraries with them. Because of this I would definitely leave the option of statically linking libqb.o (Or likely, libqb.a) into compiled programs.

All that said, replacing the current build system with make will take a bit more then is in this pull request. All of the "setup_build.sh" scripts should be replaced with Makefiles that perform the same function. I have a bit of an idea about how we might go about this, so I might take a shot at it. For now I'd just be focused on replacing the Linux scripts, replacing the Windows and OSX ones would have to come later. I'm not quite clear on how QB64 itself interacts with the build scripts though when compiling a program, so that's definitely something that needs to be looked at if we're going to be replacing the full set of build scripts.

[ghost]

@ismaell
I think this overview will provide you with enough insight to get a general idea, though I'm still new to the codebase myself and this is based solely on my own exploration:

• internal/c/qbx.cpp: The C++ source file compiled and linked to create a binary.
• internal/c/libqb.cpp: The C++ source file compiled and linked to create libqb_XXXXXX.o. When using the setup script, XXXXXX is "setup". When using the QB64 compiler, it's a string based on the version of QB64 and a set of dependencies that should be loaded to make the program work, such as sockets, fonts, etc. For OS X, libqb.mm is used instead, which simply includes libqb.cpp.
• internal/source: Contains all files that would be in internal/temp if compiling source/qb64.bas using an existing qb64 binary.
• internal/temp: A work directory containing all files that get included in internal/c/qbx.cpp to make your program work.
• source: Contains the QB64 source code for the qb64 compiler binary, qb64.bas being the "main" file.

A brief summary of the process when compiling using the Linux or OS X setup scripts:

1. Build LibQB, FreeType, etc.
2. Copy the existing C++ code from internal/source to internal/temp
3. Compile internal/c/qbx.cpp, which includes the converted files in internal/temp and links libqb_setup.o and everything else that's needed, resulting in the qb64 binary.

And here's what happens when using an existing qb64 binary to compile source/qb64.bas, or any .bas file:

1. Convert the QB64 source code to the necessary C++ code, storing the files in internal/temp.
2. Compile FreeType, etc. as necessary.
3. Compile a new LibQB object file with the necessary dependencies encoded in the filename, if a file with that name doesn't already exist.
4. Compile internal/c/qbx.cpp, which includes the converted files in internal/temp and links libqb_XXXXXX.o and anything else that is needed, resulting a binary.

Should libqb be a shared object?

I'd say no, given that QB64 currently appears to use a specially encoded filename for libqb object files. The code for it starts at line 11885 of qb64.bas. If there's a way to remove this dependency, perhaps by loading things at runtime (e.g. dlopen and LoadLibraryA/LoadLibraryW; maybe DECLARE LIBRARY code could be used here somehow), then this might be possible. I don't think it's necessarily advantageous in any way given QB64's current design.

[ghost]

@mkilgore
I've been giving this some thought since I saw this. The current script does a lot more, but it could use some work too (fantastic work btw). I've been working on this myself, abstracting some things like OS detection into shell functions.

For example, we now have a script /etc/os-release that can be sourced on many distros with /usr/lib/os-release as a fallback location. Read more about it, if you like, though only the ID and ID_LIKE items should matter (ID_LIKE=fedora means we can check for dnf or yum; debian means we can use apt-get, etc.). If neither can be sourced, then I rely on a working lsb_release in the PATH. If even that fails (I don't have it installed on Arch for example), then the manual detection using distro-specific /etc/arch-release, /etc/debian_version, etc. is attempted.

Fedora also uses dnf now instead of yum, and CentOS and RHEL will follow suit if they haven't already, but alienating users on older versions of those platforms isn't necessarily a good idea, so yum compatibility is needed. Since rpm works for querying the package db on all three, I just use rpm -qa for listing all installed packages and rpm -q for querying whether a specific package is installed, instead of dnf list installed (or is it dnf list --installed now?) or yum list installed. dnf install $pkg works just like yum install $pkg so that's the only time the difference between dnf and yum matters.

In other words, there's some stuff that could be changed since the last version of the script was completed. Maybe it is time to just use a Makefile after all? 😛

[mkilgore]

For example, we now have a script /etc/os-release

That looks great! I'm surprised that it was around back in 2012, ./script_lnx.sh says I wrote it in 2013 (Though that could just be the date of the last update). More wide adoption of /etc/os-release probably happened after I wrote that script, or else I think I would have saw it. That said, while it's probably worth keeping the weird logic in the script as a fallback in the event /etc/os-release doesn't exist, /etc/os-release definitely looks like a much saner way to do things.

I would agree the script could use a little updating. That said, I'll be perfectly honestly that I've been a solid Gentoo user for probably 3 or more years now, so I'm a bit out of the loop on what some other distros like Fedora are doing. You would likely know better then I do.

I just put a preliminary possible version of the Makefile system in this branch:
https://github.com/mkilgore/qb64/tree/make-build-system

It ended up being fairly more involved then I expected when first starting, but my build system replaces a decent amount of the logic in qb64.bas that involves the building. The only thing I don't currently have supported is DECLARE LIBRARY dependencies, but I don't think it would be too hard to make that work. Once this is fully finished, all of the build.sh, build.bat, and build.command could be removed and the logic to run them could be removed from qb64.bas, which would just be replaced with some simpler logic to run make with the correct dependencies.

To clarify how my build system works, essentially you specify the name of the program to compile in the EXE variable, and then set the dependencies it has through a bunch of DEP_* variables, for example DEP_GL and DEP_SCREENIMAGE. The logic for the dependencies was basically just ripped out of qb64.bas around lines 11735 to 11909. For Windows support, some of the dependency logic past line 11909 would need to be put into the Makefile - it wouldn't be too hard, but I didn't bother since there's no Windows support right now anyway. (Though it actually wouldn't be that hard, mingw already includes a copy of GNU make for Windows. We just need to write replacement build.mks for the build.bats). So for example, to build the QB64 source (After it was compiled into /internal/temp) this line would work:

make EXE=qb64 DEP_FONT=y DEP_ICON=y

I also added a convenience flag to the Makefile called BUILD_QB64 which takes care of copying /internal/source to /internal/temp and then building QB64 with its current dependencies. ./setup_lnx.sh was modified to simply do a make clean and then make BUILD_QB64=y after installing packages and such.

[ismaell]

@mkilgore Would you mind to work with me towards a portable Makefile?

I can extend my implementation to include all build.mk files in the tree, and those could conditionally add stuff to the build if the OS matches, that would be more generic.

[mkilgore]

@ismaell
I'd be happy too. I think it probably makes sense to work off of what I started here just because it supports more of what QB64 needs to be able to actually build any program generated by QB64 like it does now, but I'm open too ideas. I think the big thing is the optional dependency inclusion and libqb generation, which mine already does. Unfortunately there is a fair amount of duplication between the build.mk files. It would likely be possible to reduce that with some templating, but there's not that many of them so I figured it wasn't really worth it (Debugging Make templates is a horrible experience).

As far as making it portable, I'm pretty sure mine is already pretty portable, though it definitely needs testing. The idea was that eventually we could just pass OS=blah to the Makefile and that tells it what to compile. The Makefile itself already includes the correct ./build.mk files depending on OS (But the OSX and Windows ones don't yet exist). That said, the ./build.mk files I wrote are likely already generic enough to basically support all three OSs by themselves, and the few parts that aren't could be handled with a few ifeqs easily enough. Windows also needs some extra handling for some build flags. If those cases were handled, we could likely get rid of all of the os folders all together in most cases and just have a single build.mk.

If you want to try running it on Windows, you could just copy the build.mk files I already wrote into the win directories, and then set the CC, AR, and CP variables as appropriate. There should also be a RM variable, but I think I forgot to add it, I'll be making a few more changes and I'll make sure to add that. I might also see if it works with wine, IIRC QB64 can actually be built under Wine.

I think the harder part will be retrofitting this into qb64.bas. The compilation logic there is a bit screwy, and it generates scripts and then runs them to compile the source, so it's important that we ensure there aren't any weird cases we're not handling. It shouldn't be too bad, but still.

[mkilgore]

I had surprisingly quick success getting it to work on "Windows" (Though I'm just testing in Wine). It took me a bit to figure out where qb64.bas was getting all its flags from (./internal/c/makeline_win.txt) but it's not too bad. Right now everything is compiled into the /os/lnx folders, since that was the quickest way to get things working. I also add basic support for OSX, but it's untested. I think the big thing now is looking at converting qb64.bas to use make, and adding the few things my Makefile setup is missing (the inline_DATA option isn't handled, and DECLARE LIBRARY stuff isn't handled, and possibly others like icon stuff). It definitely needs a lot of testing. If we can manage that, then we could get rid of the os directories and all the build.(sh|bat|command) files.

[FellippeHeitor]

@mkilgore Hi, Matt. It's nice to see you around. Given how long ago your last contribution was I didn't even consider bringing you into this PR initially, for which I apologize. Turns out your contribution was solid enough not to require you to step back in for four years :-)

Good to have you with us.

[ismaell]

@mkilgore It uses features specific to gmake, that's what I meant, I would prefer it to build with bmake too.

A portable .mk file could include instruction for all operating systems and would look like this:

component_name-objs = file3.o file4.o
component_name-objs-osx = file1.o
component_name-objs-lnx = file2.o
component_name-objs += $(component_name-objs-$(OS))
objs-$(COMPONENT_NAME) += $(component_name-objs)

So you can do something like make OS=osx COMPONENT_NAME=y. Obviously component_name is a placeholder.

[mkilgore]

Ah, I see your point. I'm open to doing that, is there a reason for supporting bmake though? There is more GNU specific stuff then just the 'if' lines, and all the platforms QB64 supports have easy access to GNU make.

[ismaell]

@mkilgore well, just because if a dependency isn't strictly needed, then probably it shouldn't be forced.

Let me check what I can do with your implementation.

[mkilgore]

That's fair. I guess the way I see it is that we're already likely dependent on GCC anyway, so not having a dependency on GNU make doesn't really get us much. That said, I don't really feel like that's my call to make, so I'm fine with making it more standard make syntax.

Your definetely right that it could be cleaned up. I was mostly just focused on duplicating the current functionally correctly.

[ismaell]

I will copy here my comments on the commit for completeness:

---

Conditional rules could be rewritten like this: https://gist.github.com/ismaell/48fc527236888456d83a676d6c1ba9ab. But I would prefer to have a Makefile.$(OS) instead to provide those variables, it would be cleaner, and if there were little duplication, same could be done with the build.mk files...

Probably build.mk files should describe the component for all operating systems, and the main Makefile should include a Makefile.$(OS) with just the OS-specific variables.

---

@mkilgore Which way would you prefer to go?

[mkilgore]

@ismaell
OS specific Makefiles sounds fine with me, I do agree it's the cleaner way to do it. Going forward I definitely think it'll be important to have some good test cases for the weirder dependency situations, just to make sure we don't break anything.

The build.mk files in my change already work for every OS, I just haven't moved them out of the lnx folder from testing since I didn't have much time and wanted you to be able to see what I had. So we can just move those files out of the os folder, and then once qb64.bas supports using make we can throw-out the os folders completely. I also agree that there should only be on bulid.mk per 'part'. The audio one has a bunch but there should at least be a single 'decode' build.mk that includes the other files.

With that said, do we have any way to test OSX? It seems like that might be a problem we're going to run into.

@FellippeHeitor
Definitely not your fault, honestly I was surprised I still have rights on this repo. I usually try to skim all the notifications I have (I don't get that many) and this one happened to partially be code I wrote.

[FellippeHeitor]

I can provide testing on macOS. I'm on High Sierra.

[paulwratt]

Once this is fully finished, all of the build.sh, build.bat, and build.command could be removed and the logic to run them could be removed from qb64.bas ...

I agree with the process going on here, but for the sake of those systems that dont have a make but do have a G++ compatible compiler and a SH, please make a current (full/original) qb64.bas available in the main/master release somewhere, not just for posterity.

[Kroc]

Agreed with above, removing build.bat would make building a Windows executable a living hell for 99% of Windows developers.

[mkilgore]

Obviously, this project fell a bit off the radar. I don't personally do much with QB64 anymore so while IIRC the make setup I had in my repo did work, if nobody really cares for it I'm not sure there's much of a reason to continue down this path. I think the reality is that the current build system, though convoluted, works, and that's all that really matters at this point.

Agreed with above, removing build.bat would make building a Windows executable a living hell for 99% of Windows developers.

I'm not sure I understand this. We distribute GNU make with the mingw compiler, so every QB64 user already has make. With that, QB64.exe is what calls make anyway, so ideally almost nobody notices a change.

[FellippeHeitor]

Agreed and willing to close. Everybody else, thoughts?

[flukiluke]

I very much like the idea of using make to build QB64, in principle at least. However, given that this is ultimately a refactor of an already working process, I don't think I'd be able to find the time or motivation to properly check and integrate this (not at this point in time, anyway).

[ismaell]

Unfortunately, checking and testing all the distros takes a decent chunk of time, I remember that being the most annoying part of writing that script.

We could just rely on pkg-config to do that job, that would simplify a great deal.

As for making libqb.o a shared object instead, it's not a horrible idea, but it would mean that QB64 executables are not as easily redistributable. Keep in mind that unlike Windows, Linux will not look for .so files in the same directory as the executable unless you explicitly tell it to at run time (By using the LD_LIBRARY_PATH env variable). This is a bit annoying, and makes it hard to make easily portable executables that include libraries with them. Because of this I would definitely leave the option of statically linking libqb.o (Or likely, libqb.a) into compiled programs.

It's common practice to provide scripts for that, but also it's possible to embed a path into the executable (see DT_RPATH attribute).

[ismaell]

@flukiluke @FellippeHeitor I'm still interested on pursuing this; a clean buildsystem is a must for building packages.

Though, it will take me quite some time to catch up with this and other related stuff.

[alexmyczko]

@ismaell how are you progressing? I'm interested in your Makefile to create official debian/ubuntu packages of qb64. which system will you need the makefile in?

currently I didn't find a way to set the path for the 'internal' folder to be /usr/share/qb64 (or /usr/share/qb64) and be only read-only. and tell qb64 to use /tmp or ~/.cache for writing or cache files

[ismaell]

@alexmyczko not at all; life got in the way. I'll post updates here, I've not dropped the idea yet.

[FellippeHeitor]

Moved to https://github.com/QB64Team/qb64