Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Build system cleanup phase 2 #171

Closed
wants to merge 22 commits into from
Closed

Conversation

pspacek
Copy link
Contributor

@pspacek pspacek commented Oct 18, 2016

This merges all the configure scripts into one modern, including client. The end state will be one configure script with appropriate options which will allow to build only client as necessary.

Client-only build does not work right now so it broke ONLY_CLIENT option in SPEC file. I'm sending it early so we can get review and merge it sooner rather than later.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 18, 2016

This version fixes minor problems in error reporing and should have no other visible effect.

@stlaz stlaz self-assigned this Oct 19, 2016
AC_CHECK_HEADER(popt.h, [], [AC_MSG_ERROR([popt.h not found])])
AC_CHECK_LIB(popt, poptGetContext, [POPT_LIBS="-lpopt"])
AC_SUBST(POPT_LIBS)
PKG_CHECK_MODULES([POPT], [popt])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Older version of libpopt does not provide pkg-config file.
If you want to use PKG_CHECK_MODULES anyway then it would be good to fall back to older style of detection.

sh$ rpm -q popt-devel
popt-devel-1.13-16.el7.x86_64
sh$ pkg-config --list-all | grep pop

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It prints nice informational message how to work it around and it actually works on RHEL 7, so I'm not going to comlicate the build system for crappy packages.

AC_MSG_ERROR([curl not found])
fi
AC_SUBST(CURL_LIBS)
PKG_CHECK_MODULES([CURL], [libcurl])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make sense to modernize detection of unused library?
I cannot see include for any curl header file.

$git grep -E "curl/"
client/configure.ac:AC_CHECK_HEADER(curl/curl.h, [], [AC_MSG_ERROR([curl/curl.h not found])])

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is removed in later patch. I'm not going to rebase this again. Thanks for understanding.

@lslebodn
Copy link
Contributor

ACK to
Build: modernize SASL library detection
Build: modernize XMLRPC-client library detection
Build: cleanup INI library detection

It would be good to push them if you do not want to wait for review of other patches.

@stlaz stlaz added the ack Pull Request approved, can be merged label Oct 19, 2016
@stlaz
Copy link
Contributor

stlaz commented Oct 19, 2016

ACK, works for me.

@lslebodn
Copy link
Contributor

-ACK,
my comments has not been addressed yet.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 19, 2016

Justification is above, please push.

@stlaz
Copy link
Contributor

stlaz commented Oct 19, 2016

+1 to push, the comments were added to outdated diffs so I thought them resolved. They are now.

@lslebodn
Copy link
Contributor

I am sorry I still do not agree with popt change reducing 3lines to 1 is not huge simplification.
And it does not work on rhel7.2

@lslebodn
Copy link
Contributor

and the explanation for changing curl is not appropriate.
-1

@lslebodn
Copy link
Contributor

BTW.
previously it was possible to build just a client code. But after these changes configure will require to have installed even daemon dependencies If someone decide to build just a client.
Is there a reason why there is not a configure time option for such use-case.
It would skip detections of build requirements for daemon part.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 19, 2016

I consider CURL topic to be just bikesheding.

Ad POPT: RHEL is going to require explicit configuration as designed in http://www.freeipa.org/page/V4/Build_system_refactoring because you have to explicitly disable missing checks etc. anyway. At the same time you can provide correct POPT flags.

If RHEL packager disagrees we can use some auto-magic but I would rather avoid it whenever possible. For reasons above, I think that upstream version should use pkgconfig as it is preferred way for library auto-detection.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 19, 2016

@lslebodn If you read the pull request description you will notice that client-only build will be solved later on.

@MartinBasti
Copy link
Contributor

I see just bikeshading here, on IRC Petr1 agreed that this should be pushed and incrementally upgraded according our needs

@pspacek pspacek force-pushed the build-cleanup branch 2 times, most recently from 96661e7 to a7690fb Compare October 19, 2016 13:30
@lslebodn
Copy link
Contributor

I see you need to increase patch_count and not to have proper review

@lslebodn
Copy link
Contributor

BTW I am really interested in how do you plan to implement client only build.

IMHO it would be much simpler include reduced version of daemon/configure.ac (+others)
into top level configure.ac rather then creating if/else branches in huge long unified configured.ac

@ghost ghost removed the ack Pull Request approved, can be merged label Oct 19, 2016
@pspacek
Copy link
Contributor Author

pspacek commented Oct 20, 2016

@lslebodn Decision if we need client_only build is still open. We may drop it so reimplementing it would be busy work.

@lslebodn
Copy link
Contributor

My proposal would not require any reimplementation because it will reuse current solution.
And I would like to know the reasons why upstream might consider to drop client_only feature.
There are still distribution/platforms which does not have systemd which s required for server part.
And decision to drop client-only feature would not increase portability of currently fedora based project.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 20, 2016

@lslebodn Please discuss this matter on freeipa-devel list to gethigher visibility.

Please note that current client-only build is quite broken and requires heavy machinery in downstream spec file so it is hard to speak about any reusage.

@lslebodn
Copy link
Contributor

Please note that current client-only build is quite broken and requires heavy machinery in downstream spec file so it is hard to speak about any reusage.
That's the problem of ugly downstream spec file.

This is a pure upstream discussion and how to make at least part of freeIPA more portable rather then closing doors for other platforms/distributions without systemd. There are still people on debian which does not use systemd and want to use sssd and ipa-client. The current patch-set would not create more portable client code. It would would just make it much harder and would require more downstream patches which is not very upstream friendly approach.

@pspacek, it would be good if you could comment my proposal with including configure snippets from subdirectories. m4_include works like a magic and if conditions around them would make much nicer configure script.

@tiran
Copy link
Member

tiran commented Oct 20, 2016

I agree with @lslebodn . It should be possible to build only the necessary bits and pieces for an IPA client. It should be rather simple to implement a --without-server AC_ARG_WITH() in another PR. Just move server-only PKG_CHECK_MODULES and AC_CONFIG_FILES() blocks into a `if test "x$with_server" = "xyes"; then ... fi`` block.

To be clear: I'm fine with this PR. client-only installation can be added in a later PR. Since it it easy to implement we can re-add it later without much trouble.

@tiran
Copy link
Member

tiran commented Oct 20, 2016

ACK

make rpms is passing on my machine.

@lslebodn
Copy link
Contributor

lslebodn commented Oct 20, 2016

08:53 < pspacek> lslebodn: I would appreciate patch showing the nicer solution to me. I do not think it
is that easy as you claim, especially when we count in that client_only build depends heavily on spec file, which is not usable in Debian at all, which makes
client-only-purely-upstream-point completely moot.

Currently it is possible to build just a client part with following steps. Ant it has nothing to do with downstream specfile

make version-update
cd client; ../autogen.sh --prefix=%{_usr} --sysconfdir=%{_sysconfdir} --localstatedir=%{_localstatedir} --libdir=%{_libdir} --mandir=%{_mandir}; cd ..
make client-install

As you can see configure script is executed only in client sub-directory.
Here I am not interested in python part of client code.
My concerns are pure related to C-code.

But after merging everything to the one big configure script it would not be possible
without patching a master.

And you also wanted to see some patches with nicer solution
here is a POC:

AC_ARG_WITH([server],
                [AC_HELP_STRING([--with-server],
                                [Whether to build with server support [yes]]
                               )
                ],
                [],
                with_server=yes
               )
if test x"$with_server" = xyes; then
        AC_SUBST(HAVE_SERVER)
        AC_DEFINE_UNQUOTED(HAVE_SERVER, 1, [Build with server support])
        m4_include([./install/configure.ac.inc]]
        m4_include([./daemons/configure.ac.inc]]
        m4_include([./asn1/configure.acac.inc]]
fi
m4_include([./client/configure.ac])

AM_CONDITIONAL([BUILD_SERVER], [test x"$with_server" = xyes])

Or If you you can merge configure part into main configure script rather then m4_include([./client/configure.ac])

Translations are need for client as well. This move is done to remove
dependency between client and install subdirectories.

https://fedorahosted.org/freeipa/ticket/6418
FreeIPA moved to Zanata a while ago. References to Transifex were just
leftovers.

https://fedorahosted.org/freeipa/ticket/6418
Use package config instead of checking headers.
Package config is faster because it does not invoke compiler
and guarantees proper linking flags because these are provided
by package maintainer instead of hardcoded into build system.

https://fedorahosted.org/freeipa/ticket/6418
Use package config instead of checking headers.
Package config is faster because it does not invoke compiler
and guarantees proper linking flags because these are provided
by package maintainer instead of hardcoded into build system.

https://fedorahosted.org/freeipa/ticket/6418
Use package config instead of checking headers.
Package config is faster because it does not invoke compiler
and guarantees proper linking flags because these are provided
by package maintainer instead of hardcoded into build system.

https://fedorahosted.org/freeipa/ticket/6418
Use package config instead of checking headers.
Package config is faster because it does not invoke compiler
and guarantees proper linking flags because these are provided
by package maintainer instead of hardcoded into build system.

https://fedorahosted.org/freeipa/ticket/6418
Explicit check for symbols is unncecessary because libini_config >= 1.2.0
contains all the symbols we need.

https://fedorahosted.org/freeipa/ticket/6418
It turns out that default error handling prints very useful messages
which are way better than hand-made error handling without any hints.

https://fedorahosted.org/freeipa/ticket/6418
This is temporary workaround. makeapi/makeaci transitively import
ipaplatform, which will not exist before configure is executed.
On the other hand, configure requires version.m4,
which is generated by current Makefile.

This change works around this chicked-egg problem. It will disappear
when we start using top-level configure to generate top-level Makefile.

https://fedorahosted.org/freeipa/ticket/6418
This is temporary workaround necessary until we throw away the hand-made
Makefile. ipaplatform is going to be managed by configure, but configure
right now depends on version-update target in Makefile.

https://fedorahosted.org/freeipa/ticket/6418
The original approach with __path__ implemented
by 8f98fa1 broke Pylint:
We decided to resort back to symlinks as it is easiest solution
which does not break pylint in weird ways.

This commit introduces configure --with-ipaplatform option.

https://fedorahosted.org/freeipa/ticket/6418
Apparently, the docs were not updated when ipapython/platform was moved
to ipaplatform module and internals have changed.

https://fedorahosted.org/freeipa/ticket/6418
@pspacek
Copy link
Contributor Author

pspacek commented Oct 20, 2016

@lslebodn I'm really trying to explain this but I'm still not able to get the point across.

My concerns are related purely to C-code.

Please understand that IPA client consists of components in C as well from components in Python, and also non-programatic components like translation machinery etc.

We certainly can create m4 include maze and force maintainer to always use grep before he finds particular part in the build system. Unfortunatlly, even the m4 maze will not solve the problem that client-only build of C binaries simply do not constitute working IPA client. Integration with other Python components is necessary to get the client to work.

The end goal is to fold all of hand-made Makefile and SPEC file scripts to the build system, so in the end, it should help with porting to other arches - there will be just one place where changes need to be done, instead of three.

I hope that it clears up why it is not useful to insist on keeping current pieces as they were before. The design document was sent to freeipa-devel mailing list in this thread:
https://www.redhat.com/archives/freeipa-devel/2016-October/msg00134.html
Please discuss conceptual questions on the mailing list so we can get attention of other FreeIPA developers and avoid need to point people one by one to this PR.

Thank you for understanding.

@pspacek
Copy link
Contributor Author

pspacek commented Oct 20, 2016

This PR including additional patches on top have passed build + all XMLRPC tests in Jenkins:
job/build_refactoring-build-f24build_refactoring/18/

Additional commits can be found on
https://github.com/pspacek/freeipa/commits/pr159-rebase

The test included everything up to e33e00b

@lslebodn
Copy link
Contributor

lslebodn commented Oct 21, 2016

On (20/10/16 10:29), Petr Špaček wrote:

@lslebodn I'm really trying to explain this but I'm still not able to get the point across.

My concerns are related purely to C-code.

Please understand that IPA client consists of components in C as well from components in Python, and also non-programatic components like translation machinery etc.

Correct me If I am wrong. The configure script is and will be used
just for detection of build dependencies for C-code.

So here I cannot see a conflict with my proposal.

We certainly can create m4 include maze

I cannot see a reason why it would be a maze
Detection of client build dependencies will be done in main configure.ac
Detection of server dependencies would be done in daemons/configure.ac
./ipatests/man/configure.ac and ./install/configure.ac does not have any
C-related dependencies and ./asn1/configure.ac is required by client code.
IMHO ans1c/ can be moved to client/asn1c/ (but that's offtopic)

There would not be much includes as I showed in POC

and force maintainer to always use grep before he finds particular part in the build system.
maintainers do not use grep for finding build dependencies.

The most common use case is just to run configure script and
wait for reported errors. pkg-config returns nice error messages.
And then add new build dependency.

However, C related code is not changed very often and even more
C-related dependencies are added much less often.
But if new depenendecy will be added to server part then it will
be much simpler to review whether build dependency is added to the
right section if server dependencies will be in separate file.

Unfortunatlly, even the m4 maze will not solve the problem that client-only
build of C binaries simply do not constitute working IPA client.
Integration with other Python components is necessary to get the client to work.

Totally agree. But python components is not related to checking of
build time dependencies for C-code. It should be solved in different PR
IMHO, this PR should not complicate client-only build just for C-binaries.

The end goal is to fold all of hand-made Makefile and SPEC file scripts to the build system, so in the end, it should help with porting to other arches - there will be just one place
+where changes need to be done, instead of three.

Agree, but here I cannot see any conflict with my proposal.

I hope that it clears up why it is not useful to insist on keeping current pieces as they were before. The design document was sent to freeipa-devel mailing list in this thread:
https://www.redhat.com/archives/freeipa-devel/2016-October/msg00134.html
Please discuss conceptual questions on the mailing list so we can get attention of other FreeIPA developers and avoid need to point people one by one to this PR.

I read the desing page and there is mentioned that autotools
will be used for C-part as an implementation and configure
script should have the option --enable-server which default yes.

I could not find any contradiction between my proposal and desing page.

LS

@pspacek
Copy link
Contributor Author

pspacek commented Oct 21, 2016

Correct me If I am wrong. The configure script is and will be used
just for detection of build dependencies for C-code.

Maybe this is why we do not understand each other. Autotools will orchestrate the complete build including configuration of Python (and other) components. This is what I meant by Use autotools suite to orchestrate the build on the top-level in the design document. If it is confusing I'm happy to improve it, just tell me what you want to see there.

This PR starts to use configure to create symlinks in ipaplatform Python module and more things are comming. If you look at http://www.freeipa.org/page/V4/Build_system_refactoring#Configuration you will see that configure is going contain switches for Python components as well as for Javascript components. I.e. there is nothing like C-only configure anymore.

I hope it clears the intent.

@lslebodn
Copy link
Contributor

On (21/10/16 00:41), Petr Špaček wrote:

Correct me If I am wrong. The configure script is and will be used
just for detection of build dependencies for C-code.

Maybe this is why we do not understand each other. Autotools will orchestrate the complete build including configuration of Python (and other) components. This is what I meant by Use autotools suite to orchestrate the build on the top-level in the design document. If it is confusing I'm happy to improve it, just tell me what you want to see there.

I cannot see a problem here. Design page just say that you will be able to optionally build and install
python2 and/or python3. And setuptools instead of distutils will manage that part.
I cannot see any conflict with detecting build time dependencies for C-code with enabled server and without enabled server (client_only)

This PR starts to use configure to create symlinks in ipaplatform Python module and more things are comming. If you look at http://www.freeipa.org/page/V4/Build_system_refactoring#Configuration you will see that configure is going contain switches for Python components as well as for Javascript components. I.e. there is nothing like C-only configure anymore.

Design page says something else.
./configure --without-python2 --without-python3 --without-pylint --without-jslint . But I do not care about these options.

My intention is to have a simple way how to implement "--enable-server/--disable-server". Merging everything to the one big configure script will not simplify it. It will just complicate it. And the purpose of refactoring is to make things simpler and easily maintainable.

@lslebodn
Copy link
Contributor

We (lslebodn and pspacek) agree that we disagree about maintainability of client_only in one monolithic configure.ac.

But we agreed that the refactoring of build system[1] is more important. I will close my eyes and you can push the patch-set. :-)

[1] http://www.freeipa.org/page/V4/Build_system_refactoring

@pspacek pspacek added the ack Pull Request approved, can be merged label Oct 21, 2016
@pspacek
Copy link
Contributor Author

pspacek commented Oct 21, 2016

Re-adding ACK which was removed while we were "resolving" our disagreement with Lukas.

@ghost ghost added the pushed Pull Request has already been pushed label Oct 24, 2016
@ghost
Copy link

ghost commented Oct 24, 2016

Fixed upstream
master:
https://fedorahosted.org/freeipa/changeset/329f080e6ac832ffd568e79ebca9907771f8ad61
https://fedorahosted.org/freeipa/changeset/41da8732051c193166fa69cd9bc1152d39ad8720
https://fedorahosted.org/freeipa/changeset/25dab77301f9e0289b94b0a672aed5067384c8ce
https://fedorahosted.org/freeipa/changeset/b0cb6afa2308b9d96456f0355771ecbef0ca7263
https://fedorahosted.org/freeipa/changeset/6e1d777d2878de1e90e1dab4949075d31ffb0d52
https://fedorahosted.org/freeipa/changeset/5e028b59bcd3c485cc034f491a0171f26b03ce3a
https://fedorahosted.org/freeipa/changeset/927ddcb95aeb5c9bcc0cb08c5f08cecdccd0acb8
https://fedorahosted.org/freeipa/changeset/0d37619db41abddabdd313f036f453821f438c8a
https://fedorahosted.org/freeipa/changeset/881ab4edffa5e9c6c8bd8fb9762e04d776c4a573
https://fedorahosted.org/freeipa/changeset/0d7d6f3904287bb4903ffaa9d8c61e7593ae76d6
https://fedorahosted.org/freeipa/changeset/1f1648691de6687e748c5b8267f43cf196bd7cc9
https://fedorahosted.org/freeipa/changeset/fd3176a72951a2baece36620341991feb94b230a
https://fedorahosted.org/freeipa/changeset/b12da59a6cde3580f97e4ceb7303bc85857faf4d
https://fedorahosted.org/freeipa/changeset/1e0143c159134337a00a91d4ae64e614f72da62e
https://fedorahosted.org/freeipa/changeset/8acb1f37f581e92a56951c069c990c77e83f3c42
https://fedorahosted.org/freeipa/changeset/c8be979b3263723a9ab7b3c71efbe85b42d981e9
https://fedorahosted.org/freeipa/changeset/1a4f287931edf0f3a76e97875f4af622716475af
https://fedorahosted.org/freeipa/changeset/38628e46f05bde5ccc0fbb997f79364860713b48
https://fedorahosted.org/freeipa/changeset/f25c0cb0c5f309944904fd32e781c27ae7e45a62
https://fedorahosted.org/freeipa/changeset/c954d0e1ba2a9ba8e8da679bc7246788d086d976
https://fedorahosted.org/freeipa/changeset/c70a2873f8a4447f8b38ad7b8468fc78c91bbb63
https://fedorahosted.org/freeipa/changeset/3e79d8ad4aef1f62372a2169699df345348ba3e9

@ghost ghost closed this Oct 24, 2016
@lslebodn
Copy link
Contributor

On (20/10/16 10:29), Petr Špaček wrote:

@lslebodn I'm really trying to explain this but I'm still not able to get the point across.

My concerns are related purely to C-code.

Please understand that IPA client consists of components in C as well from components in Python, and also non-programatic components like translation machinery etc.

Correct me If I am wrong. The configure script is and will be used
just for detection of build dependencies for C-code.

So here I cannot see a conflict with my proposal.

We certainly can create m4 include maze
I cannot see a reason why it would be a maze
Detection of client build dependencies will be done in main configure.ac
Detection of server dependencies would be done in daemons/configure.ac
./ipatests/man/configure.ac and ./install/configure.ac does not have any
C-related dependencies and ./asn1/configure.ac is required by client code.
IMHO ans1c/ can be moved to client/asn1c/ (but that's offtopic)

There would not be much includes as I showed in POC

and force maintainer to always use grep before he finds particular part in the build system.
maintainers do not use grep for finding build dependencies.
The most common use case is just to run configure script and
wait for reported errors. pkg-config returns nice error messages.
And then add new build dependency.

However, C related code is not changed very often and even more
C-related dependencies are added much less often.
But if new depenendecy will be added to server part then it will
be much simpler to review whether build dependency is added to the
right section if server dependencies will be in separate file.

Unfortunatlly, even the m4 maze will not solve the problem that client-only
build of C binaries simply do not constitute working IPA client.
Integration with other Python components is necessary to get the client to work.

Totally agree. But python components is not related to checking of
build time dependencies for C-code. It should be solved in different PR
IMHO, this PR should not complicate client-only build just for C-binaries.

The end goal is to fold all of hand-made Makefile and SPEC file scripts to the build system, so in the end, it should help with porting to other arches - there will be just one place where changes need to be done, instead of three.

Agree, but here I cannot see any conflict with my proposal.

I hope that it clears up why it is not useful to insist on keeping current pieces as they were before. The design document was sent to freeipa-devel mailing list in this thread:
https://www.redhat.com/archives/freeipa-devel/2016-October/msg00134.html
Please discuss conceptual questions on the mailing list so we can get attention of other FreeIPA developers and avoid need to point people one by one to this PR.

I read the desing page and there is mentioned that autotools
will be used for C-part as an implementation and configure
script should have the option --enable-server which default yes.

I could not find any contradiction between my proposal and desing page.

LS

This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ack Pull Request approved, can be merged pushed Pull Request has already been pushed
Projects
None yet
5 participants