New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build system is falling apart #336

Closed
zultron opened this Issue Oct 16, 2014 · 48 comments

Comments

Projects
None yet
8 participants
@zultron

zultron commented Oct 16, 2014

The build system is becoming more and more unmaintainable.

Of course the old problems persist:

  • So complicated and contrived that whenever someone modifies it, new problems are introduced
  • The distinction between 'RIP' and non-'RIP'/system installs is the source of many problems
    • Changes that work with RIP build break in system build
    • Setting ${prefix} inside of configure.in doesn't work as expected, leading to horribleness like scripts/gen-rtapi.ini.sh.in.tmpl which must be processed three times before rendering a correct rtapi.ini file
    • Shared object files and rpath links repeatedly break
  • General inflexibility causes headaches whenever something new is introduced; it took almost a week for the first version of the UB hack that builds separate sets of modules for each thread flavor, much more time since, and that still has unresolved issues.

New observed problems:

  • Cython build issues
    • Non-RIP install: Cython shared object files like rtapi.so are correctly installed into the python path, but also into /usr/lib; rtapi_app_* can try (and fail) to load this instead of the RT module rtapi.so in some circumstances
    • The new Cython hal/cython/Submakefile is a horrible thing whose utter nastiness can't be blamed upon its author, but rather is a testament to how difficult it must have been to integrate the generation of Cython C sources and then build of those sources into the build system.
  • Installing header files into a subdirectory, such as ../include/userpci, is an exception to the usual pattern of putting everything into ../include, and took two tries to get working, and still with no general solution.

...and the list goes on.

The solution is to replace the build system. Here are the things I see need to be done:

  • Split off any pieces that we plan to split off to make this easier
    • Documentation, as noted in #100
    • Splitting out GUIs and other non-core parts; I'm unaware of current related issues in the tracker
  • Figure out what the new build system should look like
    • I have written a cmake configuration that builds much of the core already, and am partial to this
    • Another obvious alternative is automake, a natural complement to the existing autoconf config
  • Implement it
    • It is likely possible to develop the new build system alongside the existing system for minimal disruption; I've seen this done with at least cmake (FreeCAD project)
    • Fix packaging for new build system

@zultron zultron added the problem label Oct 16, 2014

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 16, 2014

Wow, things are even worse than I thought.

I take it as a point of honor to achieve separation of the documentation ala #100.

I can test bits and pieces as we split off the non-core parts but your second and third major bullets call for work above my pay grade (oh, wait, I'm retired; everything is above my pay grade :) )

The existing build system was cobbled together rather like Frankenstein's monster, and like the monster it has to go before it causes more harm. If it is true that a new build system can be developed alongside the existing system, then we have nothing to lose except the time spent on it and everything to gain if it succeeds.

Wow, things are even worse than I thought.

I take it as a point of honor to achieve separation of the documentation ala #100.

I can test bits and pieces as we split off the non-core parts but your second and third major bullets call for work above my pay grade (oh, wait, I'm retired; everything is above my pay grade :) )

The existing build system was cobbled together rather like Frankenstein's monster, and like the monster it has to go before it causes more harm. If it is true that a new build system can be developed alongside the existing system, then we have nothing to lose except the time spent on it and everything to gain if it succeeds.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Oct 16, 2014

@RunningLight, if you want to take that on, I would be extremely grateful and do anything in my power to help.

I heard a rumor that @kinsamanka might be interested in spinning out GUIs. ;)

zultron commented Oct 16, 2014

@RunningLight, if you want to take that on, I would be extremely grateful and do anything in my power to help.

I heard a rumor that @kinsamanka might be interested in spinning out GUIs. ;)

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@kinsamanka

kinsamanka Oct 16, 2014

Please show me where to start :)

Please show me where to start :)

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 17, 2014

On 10/16/2014 07:49 PM, kinsamanka wrote:

Please show me where to start :)

And the King said to the White Rabbit "Begin at the beginning and go on
till you come to the end: then stop." [Alice's Adventures in Wonderland]

I think I put my hand up for the easy part since the documentation,
although seemingly interwoven in the Makefile/Submakefile hierarchy, is
separate from the code in terms of the actual processing. In principle,
I suppose, the same could be said of the GUI source(s), at least John
seems to think so, but I'm too cowardly to get into that part of it :)

When I get bored reading Makefiles, I often try running make with some
of its cool options like --debug and --print-data-base to get an idea
what it is doing and why (best not to run with multiple jobs, e.g.,
--jobs=; the output can get confusing), although it generates a lot
of lines in a hurry. But you probably know more about this stuff than I
do already.

Regards.
Kent

On 10/16/2014 07:49 PM, kinsamanka wrote:

Please show me where to start :)

And the King said to the White Rabbit "Begin at the beginning and go on
till you come to the end: then stop." [Alice's Adventures in Wonderland]

I think I put my hand up for the easy part since the documentation,
although seemingly interwoven in the Makefile/Submakefile hierarchy, is
separate from the code in terms of the actual processing. In principle,
I suppose, the same could be said of the GUI source(s), at least John
seems to think so, but I'm too cowardly to get into that part of it :)

When I get bored reading Makefiles, I often try running make with some
of its cool options like --debug and --print-data-base to get an idea
what it is doing and why (best not to run with multiple jobs, e.g.,
--jobs=; the output can get confusing), although it generates a lot
of lines in a hurry. But you probably know more about this stuff than I
do already.

Regards.
Kent

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 17, 2014

I forgot to mention the tracing trick built into our Makefile and triggered by the option VV=1 (see head of Makefile).

I forgot to mention the tracing trick built into our Makefile and triggered by the option VV=1 (see head of Makefile).

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Oct 17, 2014

Hmm, seems we should start by figuring what to split off. The docs are a bit easier than the rest of it, does that sound right?

In my mind, CNC is one of many possible applications for Machinekit. As such, that application (or applications?) should be split off into a separate repo (or repos), be taught to build against a Machinekit system install, and of course be packaged in our Debian archives.

I'm not clear where the splitting point should be, but hope this can be done piecemeal. For example, the various GUIs like Axis, Touchy, etc. could be spun out either in a lump to one new repo or separately to several repos, depending on how much they share in common.

How does that sound so far? Any more clear idea of where exactly the splitting point would be?

zultron commented Oct 17, 2014

Hmm, seems we should start by figuring what to split off. The docs are a bit easier than the rest of it, does that sound right?

In my mind, CNC is one of many possible applications for Machinekit. As such, that application (or applications?) should be split off into a separate repo (or repos), be taught to build against a Machinekit system install, and of course be packaged in our Debian archives.

I'm not clear where the splitting point should be, but hope this can be done piecemeal. For example, the various GUIs like Axis, Touchy, etc. could be spun out either in a lump to one new repo or separately to several repos, depending on how much they share in common.

How does that sound so far? Any more clear idea of where exactly the splitting point would be?

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 17, 2014

I believe that splitting off the documentation is an easy concept and will be a rather mechanical activity. @zultron, your last comment reminds me that most of the existing documentation in the distro is specific to what you just called the CNC application and I like to call LinuxCNC/MK (e.g., LinuxCNC implemented in Machinekit). Documentation specific to Machinekit is mostly outside the machinekit repo, e.g., the material in various places that we're aiming to sweep into machinekit.io, which means that machinekit.github.io is its target repo, at least for the time being. Any detailed discussion of the documentation should probably be in #100.

As for splitting up the code base, I will take your lead. I am more the canary in the mineshaft than a miner:)

I believe that splitting off the documentation is an easy concept and will be a rather mechanical activity. @zultron, your last comment reminds me that most of the existing documentation in the distro is specific to what you just called the CNC application and I like to call LinuxCNC/MK (e.g., LinuxCNC implemented in Machinekit). Documentation specific to Machinekit is mostly outside the machinekit repo, e.g., the material in various places that we're aiming to sweep into machinekit.io, which means that machinekit.github.io is its target repo, at least for the time being. Any detailed discussion of the documentation should probably be in #100.

As for splitting up the code base, I will take your lead. I am more the canary in the mineshaft than a miner:)

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@kinsamanka

kinsamanka Oct 17, 2014

Maybe we could start with packaging first and work backwards. Once we know how things look like, we can easily split the source into individual pieces.

Maybe we could start with packaging first and work backwards. Once we know how things look like, we can easily split the source into individual pieces.

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 17, 2014

@zultron, is it fair to say the current content of ./machinekit/debian reflect your thinking at some point in time about the packaging? It's based on "machinekit" as the name of the whole enchilada:

  • machinekit-dev
  • machinekit-doc-{en|es|fr}
  • machinekit
  • machinekit-kernel
  • machinekit-posix
  • machinekit-rt-preempt
  • machinekit-xenomai

Might that in turn suggest a new packaging for the split version (with my arbitrary nomenclature for the linuxcnc app)

machinekit linuxcnc/mk
machinekit linuxcnc-mk (depends on machinekit)
machinekit-dev linuxcnc-mk-dev (?)
machinekit-doc linuxcnc-mk-doc
machinekit-kernel ?
machinekit-posix ?
machinekit-rt-preempt ?
machinekit-xenomai ?
(machinekit-specific gui package?) linuxcnc-mk-axis (depends on linuxcnc-mk)
etc.

@zultron, is it fair to say the current content of ./machinekit/debian reflect your thinking at some point in time about the packaging? It's based on "machinekit" as the name of the whole enchilada:

  • machinekit-dev
  • machinekit-doc-{en|es|fr}
  • machinekit
  • machinekit-kernel
  • machinekit-posix
  • machinekit-rt-preempt
  • machinekit-xenomai

Might that in turn suggest a new packaging for the split version (with my arbitrary nomenclature for the linuxcnc app)

machinekit linuxcnc/mk
machinekit linuxcnc-mk (depends on machinekit)
machinekit-dev linuxcnc-mk-dev (?)
machinekit-doc linuxcnc-mk-doc
machinekit-kernel ?
machinekit-posix ?
machinekit-rt-preempt ?
machinekit-xenomai ?
(machinekit-specific gui package?) linuxcnc-mk-axis (depends on linuxcnc-mk)
etc.
@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Oct 17, 2014

Great idea @kinsamanka to split out packages first. @RunningLight, forgive me for not understanding the two columns, can you clarify? A simple list might look something like this:

Machinekit

  • machinekit (core RT, HAL, ???)
  • machinekit-dev
  • machinekit-doc-{en|es|fr}
  • machinekit-{posix|rt-preempt|etc.} ('thread flavor' packages)

LinuxCNC
Here it gets fuzzy for me, but:

  • linuxcnc-mk (-base?) (common files)
  • linuxcnc-mk-dev
  • linuxcnc-mk-doc
  • linuxcnc-mk-{axis|touchy|gscreen|gladevcp|gmoccapy}
  • emcweb (@kinsamanka?)

Not important now, but also consider alternatives to linuxcnc-mk, which is nice and brief but whose meaning might not be obvious. A machinekit-linuxcnc naming scheme has clear meaning but has clear potential to annoy: machinekit-linuxcnc-touchy.

zultron commented Oct 17, 2014

Great idea @kinsamanka to split out packages first. @RunningLight, forgive me for not understanding the two columns, can you clarify? A simple list might look something like this:

Machinekit

  • machinekit (core RT, HAL, ???)
  • machinekit-dev
  • machinekit-doc-{en|es|fr}
  • machinekit-{posix|rt-preempt|etc.} ('thread flavor' packages)

LinuxCNC
Here it gets fuzzy for me, but:

  • linuxcnc-mk (-base?) (common files)
  • linuxcnc-mk-dev
  • linuxcnc-mk-doc
  • linuxcnc-mk-{axis|touchy|gscreen|gladevcp|gmoccapy}
  • emcweb (@kinsamanka?)

Not important now, but also consider alternatives to linuxcnc-mk, which is nice and brief but whose meaning might not be obvious. A machinekit-linuxcnc naming scheme has clear meaning but has clear potential to annoy: machinekit-linuxcnc-touchy.

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Oct 17, 2014

Looks to me like we're on the same wavelength. Let's abandon the table. I
can see how the side-by-side listing would be confusing.

I'm not wedded to the '-mk' nomenclature, just struggling to differentiate

  1. our LinuxCNC over Machinekit infrastructure vs the rest of Machinekit,
    and 2) our LinuxCNC over Machinekit infrastructure vs LinuxCNC Classic.
    Both issues are problematic. In the 3D printing community, for example,
    Machinekit is considered to mean LinuxCNC for BBB. And in the CNC
    community, there is as you say, touchiness.

I remember early discussions about architecture and naming. I wish we had
finished that discussion before we went public with Machinekit but that's
water over the dam now.

Oh, and I like '-base'

Looks to me like we're on the same wavelength. Let's abandon the table. I
can see how the side-by-side listing would be confusing.

I'm not wedded to the '-mk' nomenclature, just struggling to differentiate

  1. our LinuxCNC over Machinekit infrastructure vs the rest of Machinekit,
    and 2) our LinuxCNC over Machinekit infrastructure vs LinuxCNC Classic.
    Both issues are problematic. In the 3D printing community, for example,
    Machinekit is considered to mean LinuxCNC for BBB. And in the CNC
    community, there is as you say, touchiness.

I remember early discussions about architecture and naming. I wish we had
finished that discussion before we went public with Machinekit but that's
water over the dam now.

Oh, and I like '-base'

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Oct 18, 2014

Agree with the differentiation. My first impulse was simply e.g. linuxcnc-axis, sure to cause all kinds of hell.

If @kinsamanka agrees with the above list, I'd like to start splitting out the packaging.

zultron commented Oct 18, 2014

Agree with the differentiation. My first impulse was simply e.g. linuxcnc-axis, sure to cause all kinds of hell.

If @kinsamanka agrees with the above list, I'd like to start splitting out the packaging.

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@kinsamanka

kinsamanka Oct 18, 2014

Please do.

Can we just bunch emcweb with the gui at the moment? Or just leave it as is as it needs to be updated to webtalk..

Please do.

Can we just bunch emcweb with the gui at the moment? Or just leave it as is as it needs to be updated to webtalk..

@zultron

This comment has been minimized.

Show comment
Hide comment

zultron commented Oct 18, 2014

@kinsamanka, sounds fine.

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@kinsamanka

kinsamanka Oct 25, 2014

I've played with cmake for the UIs here: https://github.com/kinsamanka/machinekit-legacy-ui

Compiles cleanly, no debian installer and I haven't tested it yet :)

I've played with cmake for the UIs here: https://github.com/kinsamanka/machinekit-legacy-ui

Compiles cleanly, no debian installer and I haven't tested it yet :)

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Oct 27, 2014

Fantastic, @kinsamanka! I'll find time to look soon.

I've been working on getting the new nosetests working, refactoring some python bindings and rewriting msgd in python. With the UIs and other bits spun off, the main thing that really cramps me is the build system. I'm super excited!

zultron commented Oct 27, 2014

Fantastic, @kinsamanka! I'll find time to look soon.

I've been working on getting the new nosetests working, refactoring some python bindings and rewriting msgd in python. With the UIs and other bits spun off, the main thing that really cramps me is the build system. I'm super excited!

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Jan 15, 2015

Another example of how broken the build system is in #443. These antics would be completely unnecessary if the 'RIP' build distinction didn't exist, and $prefix were not defined in configure.in.

zultron commented Jan 15, 2015

Another example of how broken the build system is in #443. These antics would be completely unnecessary if the 'RIP' build distinction didn't exist, and $prefix were not defined in configure.in.

@machinekoder

This comment has been minimized.

Show comment
Hide comment
@machinekoder

machinekoder Jan 17, 2015

Member

$prefix is not the only variable. At some point someone has to rework the whole build system.

Just an idea, if we do not have the expertise to rework the build system using cmake or anything else maybe we could crowdfund it and hire someone capable of doing this. Maybe https://www.bountysource.com/?

Member

machinekoder commented Jan 17, 2015

$prefix is not the only variable. At some point someone has to rework the whole build system.

Just an idea, if we do not have the expertise to rework the build system using cmake or anything else maybe we could crowdfund it and hire someone capable of doing this. Maybe https://www.bountysource.com/?

@kinsamanka

This comment has been minimized.

Show comment
Hide comment
@kinsamanka

kinsamanka Jan 17, 2015

Does it mean we can remove RIP installs? :-)
On Jan 16, 2015 5:59 AM, "John Morris" notifications@github.com wrote:

Another example of how broken the build system is in #443
#443. These antics would
be completely unnecessary if the 'RIP' build distinction didn't exist, and
$prefix were not defined in configure.in.


Reply to this email directly or view it on GitHub
#336 (comment)
.

Does it mean we can remove RIP installs? :-)
On Jan 16, 2015 5:59 AM, "John Morris" notifications@github.com wrote:

Another example of how broken the build system is in #443
#443. These antics would
be completely unnecessary if the 'RIP' build distinction didn't exist, and
$prefix were not defined in configure.in.


Reply to this email directly or view it on GitHub
#336 (comment)
.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Jan 17, 2015

It doesn't seem there's any lack of expertise on the build system. @kinsamanka has already split out the UIs, I already have an old branch that builds a lot of the core stuff with cmake, and @luminize (the latest to tear out hair while modifying it) has already split off the docs. For me it's just a matter of time and priorities to start putting those pieces together and see what's left.

This is still on my list, though, and ties into the 'master daemon' project's goal of getting complete unit tests into package builds, building packages in OBS (almost working), and creating a duplicable Buildbot system so that the project doesn't depend so much on this one critical piece of infrastructure.

And YES, may the RIP install Rest In Peace!

zultron commented Jan 17, 2015

It doesn't seem there's any lack of expertise on the build system. @kinsamanka has already split out the UIs, I already have an old branch that builds a lot of the core stuff with cmake, and @luminize (the latest to tear out hair while modifying it) has already split off the docs. For me it's just a matter of time and priorities to start putting those pieces together and see what's left.

This is still on my list, though, and ties into the 'master daemon' project's goal of getting complete unit tests into package builds, building packages in OBS (almost working), and creating a duplicable Buildbot system so that the project doesn't depend so much on this one critical piece of infrastructure.

And YES, may the RIP install Rest In Peace!

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Jul 19, 2015

@zultron Where can I find your progress on cmake for the core libraries? Is it still up-to-date?
I found https://github.com/zultron/machinekit/tree/cmake and https://github.com/kinsamanka/machinekit-libnml, but they seemed somewhat out-of-date.

I've been experimenting with building machinekit with cmake as well.
You can see the results here: https://github.com/bobvanderlinden/machinekit/tree/cmake/projects

I wanted this to be somewhat easy to keep up-to-date, so I've started working on cmake in a fork of the machinekit repository. Hopefully git does a good enough job of keeping track of the changes and merges should be possible this way.

The 'subprojects' are structured as ${project}/{include,src} with a CMakeLists.txt next to the include and src directories. This means that I've moved the source and header files for the moment from ${machinekit}/src/.

All projects are included in a global CMakeLists.txt, where (for the moment) external libraries are configured, global defines added and config.h created. a bit like the original configure.ac does. Because all projects are included in the global cmake file, subprojects can link to eachother as well as expose their include-directories. CMake automatically keeps track of the dependencies and the order in which the projects(/libraries/executables) need to be build.

At the moment I only touched the core of machinekit. No docs, no guis. I just want the bare minimum to run on my desktop/laptop. That means only the simulator. I don't know exactly how Xenomai and the other real-time systems work, so for the moment those options are not there yet.

Lastly, I couldn't find a nice way to let autoconf and cmake cooperate, so I'm trying to get cmake to do everything. I'm not certain whether this is a good idea or not, but it does result in a cleaner environment to make progress (while keeping sanity). This does also mean a lot of configuration-options are gone.

It is rather incomplete at the moment, but I wanted to know whether this is indeed a direction we want to go for the core of machinekit?

@zultron Where can I find your progress on cmake for the core libraries? Is it still up-to-date?
I found https://github.com/zultron/machinekit/tree/cmake and https://github.com/kinsamanka/machinekit-libnml, but they seemed somewhat out-of-date.

I've been experimenting with building machinekit with cmake as well.
You can see the results here: https://github.com/bobvanderlinden/machinekit/tree/cmake/projects

I wanted this to be somewhat easy to keep up-to-date, so I've started working on cmake in a fork of the machinekit repository. Hopefully git does a good enough job of keeping track of the changes and merges should be possible this way.

The 'subprojects' are structured as ${project}/{include,src} with a CMakeLists.txt next to the include and src directories. This means that I've moved the source and header files for the moment from ${machinekit}/src/.

All projects are included in a global CMakeLists.txt, where (for the moment) external libraries are configured, global defines added and config.h created. a bit like the original configure.ac does. Because all projects are included in the global cmake file, subprojects can link to eachother as well as expose their include-directories. CMake automatically keeps track of the dependencies and the order in which the projects(/libraries/executables) need to be build.

At the moment I only touched the core of machinekit. No docs, no guis. I just want the bare minimum to run on my desktop/laptop. That means only the simulator. I don't know exactly how Xenomai and the other real-time systems work, so for the moment those options are not there yet.

Lastly, I couldn't find a nice way to let autoconf and cmake cooperate, so I'm trying to get cmake to do everything. I'm not certain whether this is a good idea or not, but it does result in a cleaner environment to make progress (while keeping sanity). This does also mean a lot of configuration-options are gone.

It is rather incomplete at the moment, but I wanted to know whether this is indeed a direction we want to go for the core of machinekit?

@luminize

This comment has been minimized.

Show comment
Hide comment
@luminize

luminize Jul 20, 2015

Member

Hi Bob, Welcome! (to others, I know Bob)
Work will be much appreciated on this i'm sure!

Member

luminize commented Jul 20, 2015

Hi Bob, Welcome! (to others, I know Bob)
Work will be much appreciated on this i'm sure!

@mhaberler

This comment has been minimized.

Show comment
Hide comment
@mhaberler

mhaberler Jul 20, 2015

Member

@bobvanderlinden - welcome and great you are taking a stab at this long-standing issue! it is sorely needed and definitely still a huge problem.

There have been issues filed related to that - in particular about splitting up the machinekit package into several independent ones; in particular the (g)UI's could be farmed out, but for a start, disabling them is fine

I'm happy to coach as needed - you can find me on skype mhaberler and https://gitter.im/mhaberler !

Member

mhaberler commented Jul 20, 2015

@bobvanderlinden - welcome and great you are taking a stab at this long-standing issue! it is sorely needed and definitely still a huge problem.

There have been issues filed related to that - in particular about splitting up the machinekit package into several independent ones; in particular the (g)UI's could be farmed out, but for a start, disabling them is fine

I'm happy to coach as needed - you can find me on skype mhaberler and https://gitter.im/mhaberler !

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Jul 20, 2015

Hi @luminize !
@mhaberler Yes, @luminize told me about gitter. I've added you.

Hi @luminize !
@mhaberler Yes, @luminize told me about gitter. I've added you.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Jul 22, 2015

Hi @bobvanderlinden,

You found my latest cmake branches from before Machinekit existed, pretty old!

I was waiting for the spinning off of UIs into their own git repos before thinking about starting a new build system from scratch again. Maybe that's not necessary, and it's great someone else is taking initiative.

I had one unresolved question about cmake. Kernel threads (esp. RTAI) depend on Linux's wacky 'kbuild' system. The current build system builds modules for both userland and kernel threads with the same set of rules. Accomplishing that is certainly one (of many) source(s) of ugliness in the make rules, but at least it provides the benefit of only having to define a module & its sources once, and the module is then built for all flavors. I didn't get far enough with cmake to find out whether it could generate the kbuild config. As far as I know, it could be trivial, impossible, or anything in between!

zultron commented Jul 22, 2015

Hi @bobvanderlinden,

You found my latest cmake branches from before Machinekit existed, pretty old!

I was waiting for the spinning off of UIs into their own git repos before thinking about starting a new build system from scratch again. Maybe that's not necessary, and it's great someone else is taking initiative.

I had one unresolved question about cmake. Kernel threads (esp. RTAI) depend on Linux's wacky 'kbuild' system. The current build system builds modules for both userland and kernel threads with the same set of rules. Accomplishing that is certainly one (of many) source(s) of ugliness in the make rules, but at least it provides the benefit of only having to define a module & its sources once, and the module is then built for all flavors. I didn't get far enough with cmake to find out whether it could generate the kbuild config. As far as I know, it could be trivial, impossible, or anything in between!

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Jul 22, 2015

@zultron As an experiment I had already build shmdrv using kbuild in cmake: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/rtapi/shmdrv/CMakeLists.txt
I'm not sure yet how well this translates to other modules.

Another 'edge' case is building the .proto files, which is done here: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/machinetalk/mtalk_proto/CMakeLists.txt

However I initially thought to work on this to more easily run Machinekit's simulator on my desktop/laptop (which runs ArchLinux and NixOS). I used the python API of LinuxCNC, but after the conversation with @mhaberler yesterday on Skype it seems better to focus attention to getting a client-api for haltalk+mkwrapper using protobufs. First a initial setup for Python to get a better feel for things and after that (my goal) an API for nodejs.

I still believe it would be better if the simulator could be run on other distros as well, but from what I understood it's not worth the effort. Or at least not as useful as the haltalk+mkwrapper client APIs.

@zultron As an experiment I had already build shmdrv using kbuild in cmake: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/rtapi/shmdrv/CMakeLists.txt
I'm not sure yet how well this translates to other modules.

Another 'edge' case is building the .proto files, which is done here: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/machinetalk/mtalk_proto/CMakeLists.txt

However I initially thought to work on this to more easily run Machinekit's simulator on my desktop/laptop (which runs ArchLinux and NixOS). I used the python API of LinuxCNC, but after the conversation with @mhaberler yesterday on Skype it seems better to focus attention to getting a client-api for haltalk+mkwrapper using protobufs. First a initial setup for Python to get a better feel for things and after that (my goal) an API for nodejs.

I still believe it would be better if the simulator could be run on other distros as well, but from what I understood it's not worth the effort. Or at least not as useful as the haltalk+mkwrapper client APIs.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Aug 4, 2015

Hi @bobvanderlinden, I've forgotten all my cmake, but your tests look promising. I suppose cmake macros could be used to generalize RTAPI module builds, hiding userspace/kthread build differences and simplifying module and source definitions.

Any idea how out-of-tree comps might work?

One more corner case is Cython, maybe already addressed by others, according to Google.

zultron commented Aug 4, 2015

Hi @bobvanderlinden, I've forgotten all my cmake, but your tests look promising. I suppose cmake macros could be used to generalize RTAPI module builds, hiding userspace/kthread build differences and simplifying module and source definitions.

Any idea how out-of-tree comps might work?

One more corner case is Cython, maybe already addressed by others, according to Google.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Aug 4, 2015

If you could generalize the RTAPI builds and place those abstractions in a cmake file, you could expose the cmake file in /usr/share/cmake-*/Modules/ or /usr/share/machinekit/CMakeModules. That way the build generalizations can be reused in the out-of-tree comps.
I haven't seen any projects exposing their cmake modules when it installs, but it might be a viable way to do so.

I have no experience with Cython. I'm guessing you found this project as well? https://github.com/thewtex/cython-cmake-example

If you could generalize the RTAPI builds and place those abstractions in a cmake file, you could expose the cmake file in /usr/share/cmake-*/Modules/ or /usr/share/machinekit/CMakeModules. That way the build generalizations can be reused in the out-of-tree comps.
I haven't seen any projects exposing their cmake modules when it installs, but it might be a viable way to do so.

I have no experience with Cython. I'm guessing you found this project as well? https://github.com/thewtex/cython-cmake-example

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 9, 2015

@bobvanderlinden, I'm looking at your cmake branch. I'm able to build it with no problem. Nice!

Why are many files moved from the src/ directory into projects/? Is that a cmake thing?

zultron commented Nov 9, 2015

@bobvanderlinden, I'm looking at your cmake branch. I'm able to build it with no problem. Nice!

Why are many files moved from the src/ directory into projects/? Is that a cmake thing?

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 9, 2015

@zultron I wanted to split include and src directories for each 'subproject', that's why I introduced the projects/ directory. That way each subproject would have a src/ and an include/. Each subproject would be built into a binary (.a, .so or .o) independently. It makes sure you're not #include <...>-ing files that your subproject is not depending on. CMake makes sure the right include-directories are used (only those include directories that your subproject is depending on). It also helped a bit to know which projects were depending on which other ones due to missing include files in the compile errors while working on this.

At the time I was mostly doing this to get a feel for Machinekit/LinuxCNC's structure, as it is quite big.

@zultron I wanted to split include and src directories for each 'subproject', that's why I introduced the projects/ directory. That way each subproject would have a src/ and an include/. Each subproject would be built into a binary (.a, .so or .o) independently. It makes sure you're not #include <...>-ing files that your subproject is not depending on. CMake makes sure the right include-directories are used (only those include directories that your subproject is depending on). It also helped a bit to know which projects were depending on which other ones due to missing include files in the compile errors while working on this.

At the time I was mostly doing this to get a feel for Machinekit/LinuxCNC's structure, as it is quite big.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 9, 2015

@bobvanderlinden, ah, thanks for the explanation. I'm re-learning the cmake I once knew, which was very little to begin with!

So, is the #include issue related to avoiding the need for e.g. target_include_directories()?

I'm just trying to get hal, rtapi and machinetalk compiling for now. You made a great start! I'm getting close to having Cython working, following https://github.com/thewtex/cython-cmake-example.

zultron commented Nov 9, 2015

@bobvanderlinden, ah, thanks for the explanation. I'm re-learning the cmake I once knew, which was very little to begin with!

So, is the #include issue related to avoiding the need for e.g. target_include_directories()?

I'm just trying to get hal, rtapi and machinetalk compiling for now. You made a great start! I'm getting close to having Cython working, following https://github.com/thewtex/cython-cmake-example.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 9, 2015

No, you need to do target_include_directories, but it forces you to specify all depending projects in target_include_directories, otherwise the compiler will tell you it couldn't find the #include file.

For instance for: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/emc/nml_intf/CMakeLists.txt#L19-L22
It would previously allow you to include files from posemath, nml, etc without explicitly specifying that linuxcnc subproject is depending on those other projects. It would cause errors at link-time if a dependency was missed, now it'll cause errors at compile time.

EDIT: Oh, the Cython work is great! I remember skipping that because I wasn't sure how to go about things.
Also, are you continuing from my branch? If so, there already are some parts of Machinetalk that are being build by CMake. Same goes for linuxcnchal.

No, you need to do target_include_directories, but it forces you to specify all depending projects in target_include_directories, otherwise the compiler will tell you it couldn't find the #include file.

For instance for: https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/emc/nml_intf/CMakeLists.txt#L19-L22
It would previously allow you to include files from posemath, nml, etc without explicitly specifying that linuxcnc subproject is depending on those other projects. It would cause errors at link-time if a dependency was missed, now it'll cause errors at compile time.

EDIT: Oh, the Cython work is great! I remember skipping that because I wasn't sure how to go about things.
Also, are you continuing from my branch? If so, there already are some parts of Machinetalk that are being build by CMake. Same goes for linuxcnchal.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 9, 2015

Yes, I'm continuing from your branch, which is really spiffy. There are still a bunch of chores left, but Cython is out of the way, and your ProtoBuf build works great. I predict the comp modules will require overcoming the CMake module-writing learning curve.

My goal is to get the nosetests working only. :)

zultron commented Nov 9, 2015

Yes, I'm continuing from your branch, which is really spiffy. There are still a bunch of chores left, but Cython is out of the way, and your ProtoBuf build works great. I predict the comp modules will require overcoming the CMake module-writing learning curve.

My goal is to get the nosetests working only. :)

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 10, 2015

I have a PR against @bobvanderlinden's repo that adds CMake configs for the Cython bindings and comps (not instcomps). Combined with his work, it seems like it's starting to look like a skeleton.

It looks really slick. As an example, these lines were squashed down to these. The overall logic of the configuration is so much clearer; these functions (ignore the last!) are almost readable, whereas their counterparts are spread around hal/components/Submakefile and hal/components/mkconv.sh with difficult logic.

There's still a lot of work to go, but this seems like a really good thing. We'll get to throw away the RIP build, and it looks like Machinekit can run directly out of the build tree, and no recompile needed before running make install. That means no need to compile twice for the CI system, since the regression tests can run in the tree during the package build (though it would still be best to package the tests as well). And because the system is understandable by mere mortals, it will be easy to add new parts without breaking old parts, and these mysterious dependency problems can be cleared up. Other annoyances may be solved too, such as we can stop linking libs to rtapi_app instead of linking them to the modules they belong with because the plumbing will suddenly become easy; this will clear the way to build on systems with newer linkers.

zultron commented Nov 10, 2015

I have a PR against @bobvanderlinden's repo that adds CMake configs for the Cython bindings and comps (not instcomps). Combined with his work, it seems like it's starting to look like a skeleton.

It looks really slick. As an example, these lines were squashed down to these. The overall logic of the configuration is so much clearer; these functions (ignore the last!) are almost readable, whereas their counterparts are spread around hal/components/Submakefile and hal/components/mkconv.sh with difficult logic.

There's still a lot of work to go, but this seems like a really good thing. We'll get to throw away the RIP build, and it looks like Machinekit can run directly out of the build tree, and no recompile needed before running make install. That means no need to compile twice for the CI system, since the regression tests can run in the tree during the package build (though it would still be best to package the tests as well). And because the system is understandable by mere mortals, it will be easy to add new parts without breaking old parts, and these mysterious dependency problems can be cleared up. Other annoyances may be solved too, such as we can stop linking libs to rtapi_app instead of linking them to the modules they belong with because the plumbing will suddenly become easy; this will clear the way to build on systems with newer linkers.

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Nov 10, 2015

To quote the high school kids I've been working with, this is frickin'
awesome.
On Nov 10, 2015 4:01 AM, "John Morris" notifications@github.com wrote:

I have a PR bobvanderlinden#1
against @bobvanderlinden https://github.com/bobvanderlinden's repo that
adds CMake configs for the Cython bindings and comps (not instcomps).
Combined with his work, it seems like it's starting to look like a skeleton.

It looks really slick. As an example, these lines
https://github.com/zultron/machinekit/blob/cmake-bob/projects/hal/components/Submakefile#L129-L167
were squashed down to these
https://github.com/zultron/machinekit/blob/cmake-bob/projects/hal/components/CMakeLists.txt#L15-L25.
The overall logic of the configuration is so much clearer; these functions
https://github.com/zultron/machinekit/blob/cmake-bob/projects/cmake/UseHALComp.cmake
(ignore the last!) are almost readable, whereas their counterparts are
spread around hal/components/Submakefile and hal/components/mkconv.sh
with difficult logic.

There's still a lot of work to go, but this seems like a really good
thing. We'll get to throw away the RIP build, and it looks like Machinekit
can run directly out of the build tree, and no recompile needed before
running make install. That means no need to compile twice for the CI
system, since the regression tests can run in the tree during the package
build (though it would still be best to package the tests as well). And
because the system is understandable by mere mortals, it will be easy to
add new parts without breaking old parts, and these mysterious dependency
problems can be cleared up. Other annoyances may be solved too, such as we
can stop linking libs to rtapi_app instead of linking them to the modules
they belong with because the plumbing will suddenly become easy; this will
clear the way to build on systems with newer linkers.


Reply to this email directly or view it on GitHub
#336 (comment)
.

To quote the high school kids I've been working with, this is frickin'
awesome.
On Nov 10, 2015 4:01 AM, "John Morris" notifications@github.com wrote:

I have a PR bobvanderlinden#1
against @bobvanderlinden https://github.com/bobvanderlinden's repo that
adds CMake configs for the Cython bindings and comps (not instcomps).
Combined with his work, it seems like it's starting to look like a skeleton.

It looks really slick. As an example, these lines
https://github.com/zultron/machinekit/blob/cmake-bob/projects/hal/components/Submakefile#L129-L167
were squashed down to these
https://github.com/zultron/machinekit/blob/cmake-bob/projects/hal/components/CMakeLists.txt#L15-L25.
The overall logic of the configuration is so much clearer; these functions
https://github.com/zultron/machinekit/blob/cmake-bob/projects/cmake/UseHALComp.cmake
(ignore the last!) are almost readable, whereas their counterparts are
spread around hal/components/Submakefile and hal/components/mkconv.sh
with difficult logic.

There's still a lot of work to go, but this seems like a really good
thing. We'll get to throw away the RIP build, and it looks like Machinekit
can run directly out of the build tree, and no recompile needed before
running make install. That means no need to compile twice for the CI
system, since the regression tests can run in the tree during the package
build (though it would still be best to package the tests as well). And
because the system is understandable by mere mortals, it will be easy to
add new parts without breaking old parts, and these mysterious dependency
problems can be cleared up. Other annoyances may be solved too, such as we
can stop linking libs to rtapi_app instead of linking them to the modules
they belong with because the plumbing will suddenly become easy; this will
clear the way to build on systems with newer linkers.


Reply to this email directly or view it on GitHub
#336 (comment)
.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 10, 2015

That's a great use of .cmake files! Though the python stuff is not working for me yet.

The comp.py doesn't build yet though for me. I think it's working for you because you already have comp.py in hal/utils/. Try git clean -xdf and mkdir build && (cd build && cmake ..) && cmake -C build.
I think as a general rule, no files should be generated in *_SOURCE_DIR and only in CMAKE_CURRENT_BINARY_DIR (which is inside build/). That way no extra gitignore rules are needed as the git repository stays clean.

Also, instead of using ${MACHINEKIT_SOURCE_DIR}/rtapi/rtapi/include you could use $<TARGET_PROPERTY:rtapi,INTERFACE_INCLUDE_DIRECTORIES>. That way we could move easily move things around when needed. Not sure whether that works nicely from a .cmake file though.

I'll give both changes a try. We'll see how that goes :P.

That's a great use of .cmake files! Though the python stuff is not working for me yet.

The comp.py doesn't build yet though for me. I think it's working for you because you already have comp.py in hal/utils/. Try git clean -xdf and mkdir build && (cd build && cmake ..) && cmake -C build.
I think as a general rule, no files should be generated in *_SOURCE_DIR and only in CMAKE_CURRENT_BINARY_DIR (which is inside build/). That way no extra gitignore rules are needed as the git repository stays clean.

Also, instead of using ${MACHINEKIT_SOURCE_DIR}/rtapi/rtapi/include you could use $<TARGET_PROPERTY:rtapi,INTERFACE_INCLUDE_DIRECTORIES>. That way we could move easily move things around when needed. Not sure whether that works nicely from a .cmake file though.

I'll give both changes a try. We'll see how that goes :P.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 10, 2015

@bobvanderlinden,

I fixed the comp.py problem, and cleaned up the mess in cmake/UseHALComp.cmake as well.

Do you think we should simply remove the .gitignore files in this tree?

I fixed the include directories as you suggested. There are still a few places labeled FIXME in UseHALComp.cmake and hal/cython/machinekit/CMakeLists.txt that need to be fixed once other CMakeLists.txt files are in place and I understand how UseHALComp.cmake can be distributed in a machinekit-dev package (along with or replaced by a FindHALComp.cmake file).

Thanks for the review, and thanks for holding my hand with this.

zultron commented Nov 10, 2015

@bobvanderlinden,

I fixed the comp.py problem, and cleaned up the mess in cmake/UseHALComp.cmake as well.

Do you think we should simply remove the .gitignore files in this tree?

I fixed the include directories as you suggested. There are still a few places labeled FIXME in UseHALComp.cmake and hal/cython/machinekit/CMakeLists.txt that need to be fixed once other CMakeLists.txt files are in place and I understand how UseHALComp.cmake can be distributed in a machinekit-dev package (along with or replaced by a FindHALComp.cmake file).

Thanks for the review, and thanks for holding my hand with this.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 10, 2015

@zultron Oh I just pushed a (I think) similar change on top of yours. However, it doesn't compile for me yet (your version as well as mine). There is a problem with Cython's generated c file. For instance:

It generates:

__Pyx_PyInt_From_comp_type(comp_type value);

But the compiler tells me it should be:

__Pyx_PyInt_From_comp_type(enum comp_type value);

I have tried both clang and gcc and both give me errors on those lines. (clang was a bit more helpful by suggesting the enum prefix)

It probably could be solved by a typedef in hal.h, but I'm not entirely sure that's the right way to go (as it might be incompatible with existing code that relies on hal.h). It could probably also be 'resolved' by replacing comp_type with enum comp_type after running cython. Could be worth a try, but I'll need to give that a go tomorrow or later.

Also, should we continue the conversation here (does anyone here care about the details? :P) or should we continue in bobvanderlinden#1 ?

Lastly, not holding hands at all! ;) Learning as we go.

@zultron Oh I just pushed a (I think) similar change on top of yours. However, it doesn't compile for me yet (your version as well as mine). There is a problem with Cython's generated c file. For instance:

It generates:

__Pyx_PyInt_From_comp_type(comp_type value);

But the compiler tells me it should be:

__Pyx_PyInt_From_comp_type(enum comp_type value);

I have tried both clang and gcc and both give me errors on those lines. (clang was a bit more helpful by suggesting the enum prefix)

It probably could be solved by a typedef in hal.h, but I'm not entirely sure that's the right way to go (as it might be incompatible with existing code that relies on hal.h). It could probably also be 'resolved' by replacing comp_type with enum comp_type after running cython. Could be worth a try, but I'll need to give that a go tomorrow or later.

Also, should we continue the conversation here (does anyone here care about the details? :P) or should we continue in bobvanderlinden#1 ?

Lastly, not holding hands at all! ;) Learning as we go.

@RunningLight

This comment has been minimized.

Show comment
Hide comment
@RunningLight

RunningLight Nov 10, 2015

Yes, people here care about the details.

Yes, people here care about the details.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 11, 2015

@bobvanderlinden, Ah, I didn't notice that you'd pulled my patches. I guess I should close out bobvanderlinden#1.

The top patch in the PR gets rid of the ugly stuff in UseHALComp.cmake. I can pick out those changes into a new PR if you're interested.

I think we should keep the conversation in this repo, but perhaps start a new issue specifically about CMake. Opinions?

What version of CMake are you running? I'm on Jessie:

$ dpkg-query -W cython
cython  0.21.1-1

zultron commented Nov 11, 2015

@bobvanderlinden, Ah, I didn't notice that you'd pulled my patches. I guess I should close out bobvanderlinden#1.

The top patch in the PR gets rid of the ugly stuff in UseHALComp.cmake. I can pick out those changes into a new PR if you're interested.

I think we should keep the conversation in this repo, but perhaps start a new issue specifically about CMake. Opinions?

What version of CMake are you running? I'm on Jessie:

$ dpkg-query -W cython
cython  0.21.1-1
@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 11, 2015

I rebased on your version and fixed a couple of the issues. Not sure why it would compile for you, but I needed to change this in the Cython files:
bobvanderlinden@75bac13
Because hal.h does not have typedefs for those enums:
https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/hal/linuxcnchal/include/hal.h#L202

I also had trouble with sim_axis_hardware.comp as it seemed c-functions may not be declared inside a FUNCTION:
bobvanderlinden@a8b8771

I was doing this on my laptop (with Nix), since I don't have a Debian VM here. I'll also give it a try when I'm on my desktop (with VM) again.

I rebased on your version and fixed a couple of the issues. Not sure why it would compile for you, but I needed to change this in the Cython files:
bobvanderlinden@75bac13
Because hal.h does not have typedefs for those enums:
https://github.com/bobvanderlinden/machinekit/blob/cmake/projects/hal/linuxcnchal/include/hal.h#L202

I also had trouble with sim_axis_hardware.comp as it seemed c-functions may not be declared inside a FUNCTION:
bobvanderlinden@a8b8771

I was doing this on my laptop (with Nix), since I don't have a Debian VM here. I'll also give it a try when I'm on my desktop (with VM) again.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 12, 2015

@zultron I wanted to ask, at the moment I'm building linuxcnc.a with cmake. It has a number of externs in there that any .so or executable should implement. For example: emcTrajSetHome(EmcPose). The only file that seems to implement this function is src/emc/task/taskintf.cc, which is essentially only for linuxcncsvr. When I want to build any of the other binaries that are indirectly depending on linuxcnc.a, I get undefined references.

Do you know how this works exactly? I couldn't figure this out from the Submakefiles and I wasn't sure whether there is a better way to handle this. Would it be possible to build linuxcnc.a without dangling externs?

@zultron I wanted to ask, at the moment I'm building linuxcnc.a with cmake. It has a number of externs in there that any .so or executable should implement. For example: emcTrajSetHome(EmcPose). The only file that seems to implement this function is src/emc/task/taskintf.cc, which is essentially only for linuxcncsvr. When I want to build any of the other binaries that are indirectly depending on linuxcnc.a, I get undefined references.

Do you know how this works exactly? I couldn't figure this out from the Submakefiles and I wasn't sure whether there is a better way to handle this. Would it be possible to build linuxcnc.a without dangling externs?

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 12, 2015

Never mind my last comment. I had made a dumb mistake quite a while ago for linuxcncini, which affected many other build targets. It now conforms to the old buildsystem and makes sense again.

It is now possible to build linuxcncsvr, io, milltask, halcmd and probably some others I'm forgetting. From what I found skimming through the linuxcnc/machinekit shell script, it seems these should be the executables needed to start the core of Machinekit (without GUIs, just HAL and motor). I haven't been able to start things correctly, due to there being no correct install rules and paths are still incorrect, but it should be close now I think. (crosses fingers)

Never mind my last comment. I had made a dumb mistake quite a while ago for linuxcncini, which affected many other build targets. It now conforms to the old buildsystem and makes sense again.

It is now possible to build linuxcncsvr, io, milltask, halcmd and probably some others I'm forgetting. From what I found skimming through the linuxcnc/machinekit shell script, it seems these should be the executables needed to start the core of Machinekit (without GUIs, just HAL and motor). I haven't been able to start things correctly, due to there being no correct install rules and paths are still incorrect, but it should be close now I think. (crosses fingers)

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 13, 2015

I forgot about the rtapi_app_* and rtapi_msgd. I wasn't sure what to do with all the 'flavors' of rtapi, so I just did rtapi_app_posix. The install rules for the libraries and executables are also added as well as the configs directory.

However, it doesn't seem to work yet. I tried executing the following lines in this order in separate terminals:

linuxcncsvr -ini $CONFIGS/sim/axis/axis.ini
MACHINEKIT_INI=/etc/machinekit/machinekit.ini rtapi_app_posix --instance=0 --foreground --ini=/etc/machinekit/machinekit.ini --debug --svcuuid=6d7d46ea-d2f9-459f-a9b3-17887024ea19
halcmd ping

Watching syslog (journalctl) resulted in the following error:

Nov 13 20:33:53 bob-laptop rtapi:0[30775]: rtapi_app:0: ERROR: global segment magic not changing to ready: magic=0x0
Nov 13 20:33:53 bob-laptop rtapi:0[30775]: rtapi:0: FATAL - failed to attach to global segment

I think I'm overlooking something obvious, but it's hard to tell what.
The commands above are somewhat extracted from linuxcnc.in, so there's a good chance it's still missing something. I'm wondering what the minimal set of commands/daemons is to set up HAL.

EDIT: Also if anyone wants to give this a try, it might be handy to use nix. Place default.nix into the root of the machinekit repository and use nix-build. This downloads/builds all necessary dependencies and will build and install machinekit. It does this separately from the rest of your system, so you don't have to worry about botching up dependencies of your own system. You should be able to run result/bin/linuxcncsvr afterwards.

I forgot about the rtapi_app_* and rtapi_msgd. I wasn't sure what to do with all the 'flavors' of rtapi, so I just did rtapi_app_posix. The install rules for the libraries and executables are also added as well as the configs directory.

However, it doesn't seem to work yet. I tried executing the following lines in this order in separate terminals:

linuxcncsvr -ini $CONFIGS/sim/axis/axis.ini
MACHINEKIT_INI=/etc/machinekit/machinekit.ini rtapi_app_posix --instance=0 --foreground --ini=/etc/machinekit/machinekit.ini --debug --svcuuid=6d7d46ea-d2f9-459f-a9b3-17887024ea19
halcmd ping

Watching syslog (journalctl) resulted in the following error:

Nov 13 20:33:53 bob-laptop rtapi:0[30775]: rtapi_app:0: ERROR: global segment magic not changing to ready: magic=0x0
Nov 13 20:33:53 bob-laptop rtapi:0[30775]: rtapi:0: FATAL - failed to attach to global segment

I think I'm overlooking something obvious, but it's hard to tell what.
The commands above are somewhat extracted from linuxcnc.in, so there's a good chance it's still missing something. I'm wondering what the minimal set of commands/daemons is to set up HAL.

EDIT: Also if anyone wants to give this a try, it might be handy to use nix. Place default.nix into the root of the machinekit repository and use nix-build. This downloads/builds all necessary dependencies and will build and install machinekit. It does this separately from the rest of your system, so you don't have to worry about botching up dependencies of your own system. You should be able to run result/bin/linuxcncsvr afterwards.

@zultron

This comment has been minimized.

Show comment
Hide comment
@zultron

zultron Nov 15, 2015

Hi @bobvanderlinden,

As for hal.h, I don't know why the Jessie Cython handles the enum statements and yours don't, or what the proper fix is, adding typedef in hal.h or doing what you did. I'm placing my bets with you, though. ;)

I haven't tried your branch yet. To run a minimal HAL, try realtime start, and halrun with some .hal file, maybe from the tests directory. Get debugging to stderr with export DEBUG=5; export MSGD_OPTS=-s.

Great progress! I'll help out with flavors next week, only targeting POSIX, RT_PREEMPT and Xenomai.

zultron commented Nov 15, 2015

Hi @bobvanderlinden,

As for hal.h, I don't know why the Jessie Cython handles the enum statements and yours don't, or what the proper fix is, adding typedef in hal.h or doing what you did. I'm placing my bets with you, though. ;)

I haven't tried your branch yet. To run a minimal HAL, try realtime start, and halrun with some .hal file, maybe from the tests directory. Get debugging to stderr with export DEBUG=5; export MSGD_OPTS=-s.

Great progress! I'll help out with flavors next week, only targeting POSIX, RT_PREEMPT and Xenomai.

@bobvanderlinden

This comment has been minimized.

Show comment
Hide comment
@bobvanderlinden

bobvanderlinden Nov 16, 2015

realtime and the configuration files are implemented now. However setuid still needs to be integrated somehow. Or is there a way to test things without setuid?

realtime and the configuration files are implemented now. However setuid still needs to be integrated somehow. Or is there a way to test things without setuid?

@ArcEye

This comment has been minimized.

Show comment
Hide comment
@ArcEye

ArcEye Jul 26, 2017

This discussion died.

The build system may not be the optimum in some opinions, but it does work and no-one has come forward to do the work to replace with a CMake system.

I managed to adapt it to build the Machinekit-HAL repo without too much trouble, in some ways it is best to stick with what you are familiar with.

ArcEye commented Jul 26, 2017

This discussion died.

The build system may not be the optimum in some opinions, but it does work and no-one has come forward to do the work to replace with a CMake system.

I managed to adapt it to build the Machinekit-HAL repo without too much trouble, in some ways it is best to stick with what you are familiar with.

@ArcEye ArcEye closed this Jul 26, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment