make-rules: use 'pkgdepend resolve -e' and REQUIRED_PACKAGES

[richlowe]

The pkg(5) in OI is new enough now to allow the use of 'pkgdepend
resolve -e' to vastly reduce build times at the expense of manually
maintaining a list of dependencies.

This change allows that to happen in the "new" way, using the
REQUIRED_PACKAGES list that most Makefiles already contain, rather than
the "old" way way, using a static resolve.deps which was never actively
used in OI.

[richlowe]

I haven't updated any existing packages with static dependencies in *.p5m, nor have I checked whether the REQUIRED_PACKAGES lists already in most packages are complete or appropriate for OI.

I have a vague feeling this could cause chaos when first used.

It does, however, speed the build up by several orders of magnitude in correct components.
The meat of the ips.mk change is from upstream circa 2014

[pyhalov]

This looks fine, I like this.
I have 2 questions:

1. can we avoid using -e flag while doing 'gmake REQUIRED_PACKAGES' ? (we already know the list is incomplete, if we do this)
2. we need mass rebuild to find out if anything is broken... @alarcher - can you assist in this? (no need, so far it's evident that a lot is broken)

[pyhalov]

3. This breaks components which don't have REQUIRED_PACKAGES set. And this is a rather big set of packages....
   suggestions:

1. somehow find out if REQUIRED_PACKAGES set in component's Makefile. If they are set, use -e (well, we have some packages, which don't set REQUIRED_PACKAGES, just because it's system/library, but I think it's not a big set).
2. fix REQUIRED_PACKAGES for all components :)

Another issue I see. Sometimes component contains REQUIRED_PACKAGES, which include component-provided packages. We usually exclude them, so that one can get all build requirements in case, when current component is still not built. This change will break such components (e.g. database/mariadb-101). Again, either we somehow fix this logic or add all dependencies to RQUIRED_PACKAGES....

[pyhalov]

@alarcher, @xen0l - can we populate REQUIRED_PACKAGES together with gcc-6 migration? @alarcher - as you rebuild all the packages anyway, can you add this change to your branch? Well, it'can make a list of possible conflicts much more :)

[richlowe]

We can't tell whether REQUIRED_PACKAGES is set, because it defaults to some (small) subset anyway. So it's always set to something, but probably something incomplete.

I think I would suggest a global run of gmake REQUIRED_PACKAGES as well as the full rebuild, when testing? If not, see also my answer regarding that. Nulling RESOLVE_DEPS in the problem makefiles should make this whole thing a no-op.

I think self-referential REQUIRED_PACKAGES should be ok. Have you seen problems?

emptying RESOLVE_DEPS, as we do, should make gmake REQUIRED_PACKAGES do the right thing. Are you having problems?

[pyhalov]

OK, so a) gmake REQUIRED_PACKAGES work as expected, b) we need to do mass gmake REQUIRED_PACKAGES and to set 'RESOLVE_DEPS=' in components with inter-packages dependencies...

[alarcher]

@richlowe @pyhalov I can cherry-pick this change to the gcc-6 build zone. During the gcc-6/gcc-7 rebuilds a number of components generated an empty REQUIRED_PACKAGES and would require manual intervention.
Let me know if I should run the suggested global gmake REQUIRED_PACKAGES and report the results. FYI a full rebuild would take a night on my machine.
Self-referential REQUIRED_PACKAGES cause issues for a build from scratch excluding install from an external repository like I did for gcc-6 (I started from a minimal install + build-essential).

[pyhalov]

I think we want auto-generated REQUIRED_PACKAGES added for components, which currently miss them together with gcc-6 changeset. However, we'll have to deal somehow with self-referential REQUIRED_PACKAGES. For this manual work will be necessary - identify all such components, fix REQUIRED_PACKAGES and add 'RESOLVE_DEPS='. I thought about setting RESOLVE_DEPS to REQUIRE_PACKAGES + auto-extracted package fmris, but I'm not sure it would be straightforward to extract them, for example, from -PYV manifests.

[pyhalov]

However... RESOLVE_DEPS are necessary after $(MANIFEST_BASE)-%.mogrified target was finished. After this we can just use transform to extract pkg.fmris from manifests just like REQUIRED_PACKAGES does for dependencies.

[normjacobs]

So before I integrated the original upstream changeset that makes the switch from checked in dependency files (resolve.deps) to build generated dependency files (using REQUIRED_PACKAGES macro contents), I ran a full build and updated each component Makefile to include missing REQUIRED_PACKAGES entries.  I would encourage the same to be done here.     -Norm

…

On 01/18/18 04:08 PM, Rich Lowe wrote: I haven't updated any existing packages with static dependencies in |*.p5m|, nor have I checked whether the REQUIRED_PACKAGES lists already in most packages are complete or appropriate for OI. I have a vague feeling this could cause chaos when first used. It does, however, speed the build up by several orders of magnitude in correct components. The meat of the |ips.mk| change is from upstream circa 2014 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#3883 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AG4DZO23Si6YfaIU4KHmI5LFjYA4hFpdks5tL8DPgaJpZM4Rjo4J>.

[richlowe]

Yes, it needs a full run of make REQUIRED_PACKAGES (or a hack I put together that pulls it together from the already built packages), and a full build to test it.

[pyhalov]

Was merged as part of #3014