debian: add parallel processing for building

[The-going]

Pass the parallel operation parameter to the make command if it is present in the calling dpkg-buildpackage

[petterreinholdtsen]

Can you explain why this is needed, given this from dpkg-buildpackage(1):

       -j, --jobs[=jobs|auto]
           Specifies the number of jobs allowed to be run simultaneously (since dpkg
           1.14.7, long option since dpkg 1.18.8).  The number of jobs matching the number
           of online processors if auto is specified (since dpkg 1.17.10), or unlimited
           number if jobs is not specified.  The default behavior is auto (since dpkg
           1.18.11) in non-forced mode (since dpkg 1.21.10), and as such it is always
           safer to use with any package including those that are not parallel-build safe.
           Setting the number of jobs to 1 will restore serial execution.

           Will add parallel=jobs or parallel to the DEB_BUILD_OPTIONS environment
           variable which allows debian/rules files to opt-in to use this information for
           their own purposes.  The jobs value will override the parallel=jobs or parallel
           option in the DEB_BUILD_OPTIONS environment variable.  Note that the auto value
           will get replaced by the actual number of currently active processors, and as
           such will not get propagated to any child process. If the number of online
           processors cannot be inferred then the code will fallback to using serial
           execution (since dpkg 1.18.15), although this should only happen on exotic and
           unsupported systems.

Should not the default handle this?

[The-going]

The assembly takes place using a script in which there is such a string:

# the standard error stream to the log file.
dpkg-buildpackage -b -us -j${NCPU_CHROOT:-2} 2>>\$LOG_OUTPUT_FILE
exit_status=\$?

Before these changes, the assembly goes in one thread, although 6 is specified.
After these changes, all 6 processors are involved.

[The-going]

It seems that the code is not universal.
Older OS versions do not support this.
https://github.com/LinuxCNC/linuxcnc/actions/runs/5516405256/jobs/10057729714?pr=2579#step:7:3994

[The-going]

Should not the default handle this?

rule for linuxcnc:

%:
	dh $@

override_dh_auto_configure:
	cd src && ./autogen.sh
	cd src && PYTHON=/usr/bin/python3 ./configure \
	    --prefix=/usr --sysconfdir=/etc \
	    --mandir=/usr/share/man \
	    $(configure_realtime_arg) \
	    $(enable_build_documentation) \
	    --disable-check-runtime-deps

override_dh_auto_build-arch:
	$(MAKE) PYTHON=/usr/bin/python3 -C src build-software

The default behavior is overridden and the output on the screen contains:

make[1]: Leaving directory '/root/build/linuxcnc-2.9.0'
   debian/rules override_dh_auto_build-arch
make[1]: Entering directory '/root/build/linuxcnc-2.9.0'
/usr/bin/make PYTHON=/usr/bin/python3 -C src build-software

But for htop:

%:
        dh $@

override_dh_auto_configure:
        echo "MAKE=$(MAKE)"
        dh_auto_configure -- \
                --enable-openvz \
                --enable-vserver \
                --enable-unicode \
                $(ARCH_DEPENDENT_CONFIGURE_PARAMS)

The debugging code shows the value of the MAKE variable:

make[1]: Entering directory '/root/build/htop-3.2.2'
echo "MAKE=/usr/bin/make"
MAKE=/usr/bin/make
dh_auto_configure -- \
        --enable-openvz \
        --enable-vserver \
......
make[1]: Leaving directory '/root/build/htop-3.2.2'
   dh_auto_build
        make -j6
make[1]: Entering directory '/root/build/htop-3.2.2'
make  all-am

The default behavior is executed.

[The-going]

In the latter case, the behavior will be as follows:
If the dpkg-buildpackage -jN parameter is specified, then $(PARALLEL) = N parallel tasks will be used.
If the parameter -jN is not specified, then $(PARALLEL) will be equal to the number of available processors in the system.

https://github.com/LinuxCNC/linuxcnc/actions/runs/5521735159/jobs/10070283186?pr=2579#step:7:4065

[petterreinholdtsen]

It seems that the code is not universal. Older OS versions do not support this. https://github.com/LinuxCNC/linuxcnc/actions/runs/5516405256/jobs/10057729714?pr=2579#step:7:3994

What do you mean by 'the code' here?

[The-going]

What do you mean by 'the code' here?

/usr/share/dpkg/buildopts.mk file is not exist in debian:buster
I extracted the code from this file and wrote it in the debian rules

[petterreinholdtsen]

[The-going]

`/usr/share/dpkg/buildopts.mk` file is not exist in debian:buster I extracted the code from this file and wrote it in the debian rules

Right, got it. The change look good to me, and might speed up build while increasing the load of our build hosts. It will howevery require the manual pages to be free of race conditions. I hope we are there already, but never checked. Hopefully the end result is a win-win situation. :) Did you check how this affected build times on github?

…

-- Happy hacking Petter Reinholdtsen

[The-going]

Did you check how this affected build times on github?

This pull request was going into two threads, as it is specified. And the build time corresponds to the time of my desktop machine.
https://github.com/LinuxCNC/linuxcnc/actions/runs/5521735159/jobs/10070283186?pr=2579#step:7:4065
23m 39s

[The-going]

It will howevery require the manual pages to be free of race conditions.

@petterreinholdtsen Did you mean this page?
building-linuxcnc

or
$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src manpages

P.S:

........
   debian/rules override_dh_auto_build-indep
make[1]: Entering directory '/root/build/linuxcnc-2.9.0'
/usr/bin/make -j6 PYTHON=/usr/bin/python3 -C src manpages
make[2]: Entering directory '/root/build/linuxcnc-2.9.0/src'
make[2]: Nothing to be done for 'manpages'.
.......

[The-going]

After watching the build process and reading the source code, I realized that not everything is as clear as it seemed at the very beginning. To properly parallelize the build process, changes are needed at earlier stages.

I think I can do it in a rather elegant way.

I will wait a few days in case someone wants to leave hints or recommendations and then close this pull request.

[petterreinholdtsen]

[The-going]

> It will howevery require the manual pages to be free of race conditions. @petterreinholdtsen Did you mean this page?

Nope, I just mistyped. I mean it might need manual _changes_ to get rid of race conditions in the build system.

To properly parallelize the build process, changes are needed at earlier stages.

What early stage do you have in mind here? Note, I believe your change should go in, but suspect more changes might be needed to take full advantage of it.

…

-- Happy hacking Petter Reinholdtsen

[The-going]

Note, I believe your change should go in, but suspect more changes might be needed to take full advantage of it.

From my observations of the build process in this pull request:

override_dh_auto_build-arch:
	$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src build-software

This section is working correctly. In my case -j6 is a 6-fold increase in performance
The manual pages, in English, are assembled in this section in parallel and look correct.

This section works strangely:

override_dh_auto_build-indep:
ifneq "$(enable_build_documentation)" ""
	$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src manpages
	$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src translateddocs
	$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src docs
endif

This line doesn't work because it's already done:
$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src manpages
Next comes the work for a separate language and document in one stream
and the assembly of the final pdf document in several parallel streams.

So that this can be parallelized and at the same time ensure the assembly
of a single document for the target language in one stream, the assembly
algorithm can be changed. To do this, I need to work with Makefile, submakefile.
It is this earlier stage that I had in mind.

But it will take extra time.

Info:
Build time for QEMU arm64 machine on x86-64:
A complete reassembly of a package with documents into one stream takes 500 minutes. (6 CPU)
Complete reassembly with this PR 266 minutes (6 cpu)
Assembly without documentation 45 minutes (6 CPUs)

[petterreinholdtsen]

[The-going]

This section works strangely: ```make override_dh_auto_build-indep: ifneq "$(enable_build_documentation)" "" $(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src manpages $(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src translateddocs $(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src docs endif ``` This line doesn't work because it's already done: `$(MAKE) $(PARALLEL) PYTHON=/usr/bin/python3 -C src manpages` Next comes the work for a separate language and document in one stream and the assembly of the final pdf document in several parallel streams.

Right. I can imagine the dependencies are not quite right for the docs and translations. The man pages need to be generated before PO and POT files are processed (translateddocs), to ensure the generated man pages can be translated.

So that this can be parallelized and at the same time ensure the assembly of a single document for the target language in one stream, the assembly algorithm can be changed. To do this, I need to work with Makefile, submakefile. It is this earlier stage that I had in mind.

Make sense to me. :)

But it will take extra time.

Thank you for looking at this. :) -- Happy hacking Petter Reinholdtsen

[The-going]

I can imagine the dependencies are not quite right for the docs and translations. The man pages need to be generated before PO and POT files are processed (translateddocs), to ensure the generated man pages can be translated.

Do you mean this when you wrote that the man pages should be collected in one stream?

---

In my work I am guided by the documentation: Debian New Maintainers' Guide
The conclusion follows from it: if the debian/rules file is as simple as a small coin, then everything is fine.
If not, it indicates problems in the Makefile.

The dh_make command generated the following:

DPKG_EXPORT_BUILDFLAGS = 1
include /usr/share/dpkg/default.mk

We have this part looks like this:

linuxcnc/debian/rules.in

Line 16 in 38943cf

include /usr/share/dpkg/pkg-info.mk

This is the first mistake. Not all necessary variables will be exported and they have to be explicitly specified.
As a result, when dh_COMMAND is redefined, we need to register the variables needed in this particular
case in each command. Otherwise, this command does nothing because it is executed in the root directory not in src.

linuxcnc/debian/rules.in

Lines 59 to 66 in 38943cf

	override_dh_auto_clean:
	dh_auto_clean
	py3clean .
	if [ -r src/Makefile.inc -a -r src/config.status ]; then $(MAKE) -C src clean -s; fi
	rm -f Makefile.inc
	rm -f src/config.log src/config.status
	rm -f $(for i in $(find . -name "*.in"); do basename $i .in; done)
	dh_clean

Only 62 and 64 lines will be executed here.
But all these actions already exist in the Makefile, and it seems strange. ....

@petterreinholdtsen
Should I continue in this pull request or make a separate PR and comment on each step?