Cargo-make to replace CMake on CXX projects (and others)?

[Kreyren]

I am rewriting Paludis https://gitlab.exherbo.org/paludis/paludis and using CMake on it is currently as painful as it can possible be, because it is:

1. Scripting language is featureless and implementation requires difficult to read code

defining almost everything as strings - Which to me is painful to use and writing in it feels like adding work instead of making the work easier due to the terrible syntax that requires everything to be longer then it needs to be alike:

Code example

# FIXME-DOCS(Krey)
# FIXME-QA(Krey): Unreadable garbage
if(${PROJECT_NAME}_DOWNSTEAM STREQUAL "")
(message STATUS "No color-scheme was selected, skipping...")
elseif(NOT ${PROJECT_NAME}_DOWNSTEAM STREQUAL "")
foreach(downsteam IN_LIST ${PROJECT_NAME}_DOWNSTREAM)
# Mokleus
if(${downstream} STREQUAL mokleus-0)
if(${PROJECT_NAME}_COMPONENT_PBIN)
set(${PROJECT_NAME}_DOWNSTREAM_MOKLEUS_0 ON)
endif
set(${PROJECT_NAME}_DOWNSTREAM_MOKLEUS_0 ON)
elseif(${downstream} STREQUAL pbin-mokleus-0)
if(${PROJECT_NAME}_COMPONENT_PBIN)
set(${PROJECT_NAME}_DOWNSTREAM_MOKLEUS_0 ON)
endif
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_MOKLEUS_0 ON)
# Exherbo
elseif(${downstream} STREQUAL exheres-0)
set(${PROJECT_NAME}_DOWNSTREAM_EXHERES_0 ON)
elseif(${downstream} STREQUAL pbin-exheres-0)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_EXHERES_0 ON)
# Gentoo
# FIXME-QA(Krey): This is unreadable garbage
elseif(${downstream} STREQUAL gentoo-0)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_0 ON)
elseif(${downstream} STREQUAL pbin-gentoo-0)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_0 ON)
elseif(${downstream} STREQUAL gentoo-1)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_1 ON)
elseif(${downstream} STREQUAL pbin-gentoo-1)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_1 ON)
elseif(${downstream} STREQUAL gentoo-2)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_2 ON)
elseif(${downstream} STREQUAL pbin-gentoo-1)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_1 ON)
elseif(${downstream} STREQUAL gentoo-3)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_3 ON)
elseif(${downstream} STREQUAL pbin-gentoo-3)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_0 ON)
elseif(${downstream} STREQUAL gentoo-4)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_4 ON)
elseif(${downstream} STREQUAL pbin-gentoo-4)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_5 ON)
elseif(${downstream} STREQUAL gentoo-5)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_5 ON)
elseif(${downstream} STREQUAL pbin-gentoo-5)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_5 ON)
elseif(${downstream} STREQUAL gentoo-6)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_6 ON)
elseif(${downstream} STREQUAL pbin-gentoo-6)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_6 ON)
elseif(${downstream} STREQUAL gentoo-7)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_7 ON)
elseif(${downstream} STREQUAL pbin-gentoo-7)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_7 ON)
# In case 'gentoo' was used to use all gentoo downstream classes
elseif(${downstream} STREQUAL gentoo)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_0 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_1 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_2 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_3 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_4 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_5 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_6 ON)
set(${PROJECT_NAME}_DOWNSTREAM_GENTOO_7 ON)
if(${PALUDIS_NAME}_COMPONENT_PBIN)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_0 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_1 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_2 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_3 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_4 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_5 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_6 ON)
set(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_7 ON)
endif
# Dummy
elseif(${downstream} STREQUAL PALUDIS-1)
set(${PROJECT_NAME}_DOWNSTREAM_PALUDIS_1 ON)
else()
(message FATAL_ERROR "Dowsntream class '${downstream}' stored in variable '${PROJECT_NAME}_DOWNSTREAM' is not implemented, consider submitting a tracking and/or merge request")
endif()
endforeach()
else()
message(FATAL_ERROR "UNEXPECTED")
endif()

# Mokleus
option(${PROJECT_NAME}_DOWNSTREAM_MOKLEUS_0 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_MOKLEUS_0 "FIXME" OFF)
# Exherbo
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_EXHERES_0 "FIXME" OFF)
# Gentoo
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_0 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_0 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_1 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_1 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_2 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_2 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_3 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_3 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_4 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_4 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_5 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_5 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_6 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_6 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_GENTOO_7 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_GENTOO_7 "FIXME" OFF)
# Paludis
option(${PROJECT_NAME}_DOWNSTREAM_PALUDIS_1 "FIXME" OFF)
option(${PROJECT_NAME}_DOWNSTREAM_PBIN_PALUDIS_1 "FIXME" OFF)

To get sensible processing time as defining these into a function and using custom function to allow switch statement seems to increase the processing time.

2. Inefficient

• By design it's used as an interpreter to generate makefiles (probably failed abstraction) and it's generation is taking too long..
  • 0m7.605s to generate the build directory
  • 23m16.71s to compile the source code where based on my most likely inaccurate measurements at least 8 min is taken by cmake to just generate the makefiles for compilation

Measurements are on 4thread @ 2.701Ghz CPU with SO-DIMM DDR3 RAM

3. Culturary unindeal

This may sound silly, but i am keep being blamed by the OSS community for making my software to be painful to contribute to and only in english so i like to add functions alike:

case "$LANG" in
	cz-*|sk-*) printf '%s\n' "Ahoj!" ;;
	hu-*) printf '%s\n' "Szia!" ;;
	en-*|*) printf '%s\n' "Hello!"
esac

So that the output is in the language of the end-user while taking in mind mental illnesses like tokophobia https://en.wikipedia.org/wiki/Tokophobia to remove mentioning of birth with replacement string.

---

Cargo-make is already able to be used for this through #473 to inject functions to handle the runtime, but the implementation is 'ugly' (hard to read, requires duplicate code, etc..)

So this issue proposes adding a scripting language to cargo-make that would allow it to:

1. Inject variables in the source code

For example in cmake there is option() that allows parsing argument from cli e.g:

# cmake -DSOMETHING="whee" path/to/source
option(SOMETHING "Variable to set something" OFF) # Sets variable 'SOMETHING' to 'OFF' if not set already

This can be already done via cargo-make through --env, but it lacks the variable assigned string which is minor.

2. Cargo-make lacks library handling

In cmake there are libraries that can be fetched in the configuration e.g. GNUInstallDirs https://cmake.org/cmake/help/v3.0/module/GNUInstallDirs.html that provides variables alike SYSCONFDIR that stores /etc to comply with File System Hierarchy v3.0 standard

3. Possibly more things..

---

Proposal

In theory the ion from the Redox project https://github.com/redox-os/ion could be used for the scripting and just allowing to provide function handling while keeping Makefile.toml to handle the targets that call functions from the scripting file e.g. makersLists.txt (probably a better name..)

Would this be in the scope of cargo-make?

[sagiegurari]

thanks for the detailed explanation and examples.
personally i always use the tool for the platform, so i also use cmake for c/c++ but i know there are general build systems out there like buck/bazel/....

cargo-make in general can be used to run any flow and for rust gives much extra power.
but for other platforms it doesn't have any extra insight to help as it does for rust.

it doesn't mean it won't work, but you would have to do things manually.
it does come with few stuff that can help you kick start it

• load_script - this enables you to share common makefiles across projects. so you can build all the cxx specific logic, put it in some repo for example, and load it from any cxx project you have using the load_script.
  Since load_script can run duckscript, you can also very easily do cross platform http get or ftp get if git clone is not good enough.
• duckscript - that's the currently only embedded language that cargo-make knows. it knows to do a LOT of stuff and if needed can be improved to include additional commands.
  But.... you can always create a task with a lot of duckscript functions to contain the logic and then just extend/use that task and in your extended task script call those functions with actual values.
• cli variables - ya, i have the --env and with duckscript you get those automatically as script variables. so if you use duckscript its available like any other variable (not just as env var).

can you check it out first before we think of better solution?
i'm trying to see if we do actually have a gap or not.
having a cxx makefile to help cxx projects use cargo-make can be a nice open source project on its own.

[Kreyren]

load_script - this enables you to share common makefiles across projects. so you can build all the cxx specific logic, put it in some repo for example, and load it from any cxx project you have using the load_script. @sagiegurari

The issue with cmake is that it builds the makefiles per platform and the makefiles being a limiting factor atm where i want it to build on as many turing complete systems as possible including non-standard to the reasonable degree -> Assuming sourcing makefiles not a sufficient solution?

duckscript - that's the currently only embedded language that cargo-make knows. it knows to do a LOT of stuff and if needed can be improved to include additional commands.
But.... you can always create a task with a lot of duckscript functions to contain the logic and then just extend/use that task and in your extended task script call those functions with actual values. @sagiegurari

On quicklook duckscript seems to be too limited for the expected usecase, but i am happy to be proven otherwise, see scripting language requirements below.

---

DISCLAIMER: i didn't fully identified the code yet (around 60% identified), but i have a good idea what it is doing in short it's database handler that executes scripts with provided built-in API to provide packages for target system.

So i've created a stub commit in https://github.com/Kreyren/CM-Paludis intended for discussion (this is a package manager currently designed for x86_64-pc-linux-{gnu,musl} target)

With ultimate goal to compile e.g. GNU emacs using the default exheres eapi (downstream library used for processing of the package) using client cave https://github.com/Kreyren/CM-Paludis/tree/master/clients/cave using cargo-make handled compilation.

Cave depends on following libraries for runtime: https://github.com/Kreyren/CM-Paludis/blob/master/CMakeLists.txt#L18

with original code at http://gitlab.exherbo.org/paludis/paludis (Fork adapted to allow non-cmake management)

---

TODO list

1. We need method to create static and dynamic libraries depending on file type on demand

This seems to be possible using custom logic through load_script

2. Parsing options inside the source code prior to compilation

also load_script

3. Specifying executables for compilation

pre-defined logic overwrittable through [env] ?

4. Method to run tests

low-priority atm

load_script ?

5. Method to run benchmarks

low-priority atm

load_script ?

Requirements for the scripting language

DISCLAIMER: This are observed in relation to rustlang, cmake, posix sh and lisp -- Are probably incomplete and require brainstorm

Conditionals

• If statements
• switch

Loops

• for
• while

Linting/environments

Context: The major issue with ion that i have atm is that it doesn't have linting and the peer-review process has to be done through upstream that is taking time and is unreasonable to believe that contributors would go through this process willingly -> Ideally expecting a way to automate the review process for the contribution review through linting and tools like valgrind, indent, etc.. where less work reviewing i do the better.

• Making programming not a pain that requires sherlocking the code for missing semi-colon or alike

• Showing helpers to functions e.g. IDE for C/Rustlang will show helper pop-ups when hovering over functions

• In case the scripting language is complicated -> Show data-type helpers
  

RegEx

PCRE ideally or something embedded that allows reliable text manipulation

Comments

Essential for my project where preferred are C-style comments expected to be interpreted as in-code documentation that can ideally be formatted into rustlang-style library documentation.

Functions

Ideally something sane that allows specifying arguments and limits

Efficient

Expecting efficiency greater then POSIX sh and quicklisp

Readability

I prefer to make my code as readable as possible meaning:

• Avoiding long lines of code that can be interpreted in shorter way
• Avoiding duplicate code
• Documenting everything in-code

Handling of libraries

From my observation the most limiting factor on C is the lack of library handling that makes them more likely to be deprecated or recreated which is handled on rustlang through crates.io preferrable for scripting language as well

Optional

Method to create tests and benchmarks per function would be preferred to reduce the amount of bugs

Method to redefine the programming language for the task welcomed e.g. lisp interpreted into emacs lisp to be used in scripting in GNU Emacs

---

FWIW i like more the idea of making a scripts that compile the code that can be then put in a Makefile as code quality, reliability and readability is a concern.

[sagiegurari]

• load scripts - i just meant to use that to load the right makefile from remote to get all the functions that allow for c/cpp compilation. the project specific logic would still be in project specific makefile.

• lang features (answering in context of duckscript)

  • if/switch - you have if/elseif/else
  • for/while - you have for-in
  • lint - nope
  • regex - specific commands have regex support, such as: glob_array and split string
  • comments - bash like comments
  • functions - supported
  • Efficient - executed by rust, but naturally not as performant and not a goal of the language. having said that, its fast.
  • Readability - similar to bash redability
  • Libraries - not at the moment. requires rust development.
  • Tests - built in support for testing, including assert commands and ability to run a test file or test directory (recursive). thats actually how i test duckscript internal sdk.
  • Compilation - well.... its a scripting language, but you could take the script and the runner and compile to an executable. I do that for internal projects.

[Kreyren]

load scripts - i just meant to use that to load the right makefile from remote to get all the functions that allow for c/cpp compilation. the project specific logic would still be in project specific makefile. @sagiegurari

Can you elaborate how? I think you are proposing to maintain multiple makefiles used depending on used kernel which would require a duplicate code assuming the complexity of the CXX repo that requires compiling of libraries from X amount of files?

---

duckscript noted

---

FWIW my current proposal would be to add optional API to cargo-make that can be used as a replacement for cmake that doesn't generate the Makefiles and can be used across multiple programming languages.

I am currently abstracting a different project that is expected to be rewritten into multiple programming languages and it's components compiled based on benchmark results so i would be interested in cooperating on this if it's in the scope of this project assuming that it would provide an API to multiple scripting/programming languages to be used on demand.

[sagiegurari]

@Kreyren i just published a new repo: https://github.com/sagiegurari/cargo-make-tasks/
which will hold makefiles for different langs/envs and can be used to pull these tasks by any project (see usage in that repo).

for now i just created 1 makefile (cmake) for my cmake based c projects, but its worth checking what i did there to see if it can maybe give you ideas for your usecase.

[Kreyren]

I like the idea and i will for sure it it for a reference, but i think it should be part of this repo e.g. using examples and this issue rather wanted to replace cargo-make with cmake (i want to avoid using voodoo lang to write C++)

[sagiegurari]

@Kreyren see #542
you can now define env vars via glob paths (with optional gitignore support) which i think is another important tool for this track

[Kreyren]

I am confused what is this doing, is this using a python file to define environment variables?

FWIW i dont want to make software that during a runtime reads a lot of env vars as that would be unsafe, but i like the idea of using these during a build.

Maybe this could be implemented so that it expects a standardized file whoch sets a lots of variables e.g. how linux is configured ?

[sagiegurari]

no, its an example on creating env vars with glob path

[sagiegurari]

closing this one as i think duckscript + cmake tasks examples i gave is a good start.