New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't add generated files to the tarball during dist, needed for gtk-doc and tracker #2166

Open
hadess opened this Issue Aug 9, 2017 · 15 comments

Comments

Projects
None yet
8 participants
@hadess

hadess commented Aug 9, 2017

This would make it easier to install documentation without having to compile the library. library.gnome.org 's software doesn't integrate with a C-I and expects generated gtk-doc help files to be available in the distribution tarballs.

@nirbheek nirbheek changed the title from Add a way to force include gtk-doc output in dist to Add a way to invoke run_target()s during dist, needed for gtk-doc html documentation Aug 9, 2017

@jpakkane

This comment has been minimized.

Member

jpakkane commented Aug 9, 2017

Source is source and generated files are generated files and mixing the two will not be supported. If you want to do this you need to generate a custom dist target generator or something similar. The proper fix is to fix the CI pipeline to not require prepackaged files. This makes it also support building directly from Git without interim dist files.

@tp-m

This comment has been minimized.

Member

tp-m commented Aug 9, 2017

Might just as well check the html into git then tbh and update from time to time ;)

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 9, 2017

Why make releases at all then? Just tell people to use the git tag.

The answer of course is that the release tarball is not always the same as the git tag. Some people do need to generate OS-independent files for various reasons (reduced build time, reduced build dependencies, and so on) and distribute them with the tarball.

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 9, 2017

FWIW, I do agree that there is a line to be drawn somewhere. For instance, I don't think we should allow other types of targets to be run as a part of dist because it can and will confuse meson (think, stale glib-mkenums enumtypes in the source tree). But we do need a way to define targets that will be run as part of dist, and run targets are a good choice, IMO.

@TingPing

This comment has been minimized.

Member

TingPing commented Aug 9, 2017

This was discussed on the mailinglist also but doing this for gtk-doc was just a workaround for poor infrastructure on GNOMEs part and with better CI and hosting potentially coming this workaround might no longer be needed in the not-so-distant future?

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 9, 2017

I did get that impression from the mailing list, but it requires someone to do the work first and/or wait for the Gitlab transition. Both of those are suboptimal, and in the meantime people are putting in some really hacky workarounds in their projects. I think we can only hope that it gets resolved one way or another before the next GNOME release. :)

@hadess

This comment has been minimized.

hadess commented Aug 9, 2017

Might just as well check the html into git then tbh and update from time to time ;)

You should be worried that the "current stable" version of GStreamer's docs on developer.gnome.org has 1.2 from around 2014 (for reference, current is 1.12), because of this exact problem.

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 9, 2017

You should be worried that the "current stable" version of GStreamer's docs on developer.gnome.org has 1.2 from around 2014 (for reference, current is 1.12), because of this exact problem.

That seems unrelated because GStreamer still has Autotools as its primary build system, our tarballs have gtk-doc documentation, and the Meson build files were only added in 1.10.

@tp-m

This comment has been minimized.

Member

tp-m commented Aug 9, 2017

Yeah, I have no idea why that is. Tried to find out a few times, but no one could tell me who to poke or what the reason might be. I think that's unrelated though :)

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 13, 2017

As another datapoint, Tracker needs to include generated files in the dist tarball, and changing that will involve difficult code changes, see comment 4 on this bug.

Background: Tracker automatically tracks when its parser code changed by storing the SHA1 of the last commit to the parser code in a header so that it can migrate local databases whenever needed. This works fine in git, but doesn't work in tarballs.

<garnacho> nirbheek: background: I need to track changes over a group of files, so users get stuff triggered on their local databases when running newer/older trackers
<garnacho> this takes seconds to minutes, so I'd rather not do it on every new version

The autotools build adds that header to EXTRA_DIST so it gets included in the tarball. Not every tracker release touches the parser code, so doing the update on every release is going to be painful for users.

I think we need a mechanism to have targets that are:

  1. Only run when building from code in version control
  2. Can be run during dist and included in the generated tarball

run_target()s are probably not a good choice for this because of (1), which implies being built during compilation. We could either extend custom_target()s or create a new target called dist_target() or similar.

@nirbheek nirbheek changed the title from Add a way to invoke run_target()s during dist, needed for gtk-doc html documentation to Can't add generated files to the tarball during dist, needed for gtk-doc and tracker Aug 13, 2017

@jpakkane

This comment has been minimized.

Member

jpakkane commented Aug 13, 2017

We could either extend custom_target()s or create a new target called dist_target() or similar.

Or add_dist_script which behaves similarly to add_install_script.

@nirbheek

This comment has been minimized.

Member

nirbheek commented Aug 13, 2017

That would solve it for Tracker, but would also duplicate code. Maybe we should also have add_dist_target() which takes a custom_target() or a run_target()?

@ignatenkobrain

This comment has been minimized.

Member

ignatenkobrain commented Aug 27, 2017

I would say that it would be very handy to declare some generated files (be that gtk-doc or a2x manpages), people should be able to include those in dist tarball.. This really saves a lot of time for building such projects because stuff like sphinx pulls texlive.. But we need to have really good design for such things..

@inigomartinez

This comment has been minimized.

Contributor

inigomartinez commented Jan 9, 2018

This also happens withd docbook documentation created by gdbus-codegen. NetworkManager for example, distributes these docbook files, but since these are generated they are not added to the distributable file.

@smcv smcv referenced this issue Mar 5, 2018

Open

git version #688

@smcv

This comment has been minimized.

smcv commented Mar 5, 2018

Having a way to add files to dist tarballs would also resolve #688 by providing what git-version-gen needs, I think (specifically, it wants to add .tarball-version to the dist tarball so that the dist tarball can know its own version without .git being available).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment