amake mixes up packages when processing

[hwiedPro]

Hey, what to do to prevent amake from mixing the build processes of packages:

  configuring CMake for control/rbdl
  configured CMake for control/rbdl
  building external/eigency [progress not available]
  built external/eigency
  building control/rbdl
  built control/rbdl
  installing external/eigency
  installing external/eigency

parallel=1 is already activated.

[doudou]

There's no way.

Parallel=1 only means that only one of the phases is processed at once, but you can't control the phase order.

If you have to make sure the build of two packages is ordered, you must add a dependency between them.

[hwied]

That's suboptimal, as my process seems to forget the PYTHONPATH during the phases of the other packages.

external/eigency is defined as a python_package that calls Rock.activate_python_path(pkg).
Nevertheless in the case above it fail in install phase, with the error that it can't be installed to install/lib/python3.6/site-packages/, because this directory is not on the PYTHONPATH, which currently is empty.

To make the packages dependent wouldn't make sense as they do not depend on each other.

Installing only the external/eigency package works fine...

Do you have an idea what happens there?

[doudou]

That's suboptimal, as my process seems to forget the PYTHONPATH during the phases of the other packages.

I can't be 100% sure what I believe is the problem is the problem. Anyways, Autoproj does not "forget" environment variables, but it only propagates variables from dependencies to the package that depend on them (if you need a package's folder to be in the envvar of another, it usually means that there is a dependency between the two that needs to be made explicit)

However, I don't think it is what happens here. Given what you told me, it seems to me that eigency refuses to install itself somewhere if that somewhere is not part of the PYTHONPATH. This might be because its install process is non-standard (and it requires to run Python code it just installed), but I'm guessing it's not that.

If I were you, I would look into it and find out why this behavior exists (for the record, it's a very rare behavior, and usually very misguided one).

The conflict is that Autoproj does not add non-existent directories to *PATH variables because, you know, they don't exist.

You could create the folder before the installation by adding a post_import block which creates the directory

python_package "external/eigency" do |pkg|
  pkg.post_import do
     # create the site-package directory, see code in the python helpers ...
  end
end

The other solution (which would be the one I would take personally) is to patch the package itself, or propose a pull request to the package's maintainer to remove this behavior.

[hwiedPro]

I tired it now with:

def python_package(name, workspace: Autoproj.workspace)
    package_common(:python, name, workspace: workspace) do |pkg|
        bin, version, sitelib = Rock.activate_python_path(pkg)
        pkg.depends_on 'python3'
        pkg.post_import do
           # ++ the following 3 lines are new
            if !File.exist?(sitelib) then
                FileUtils.mkdir_p(sitelib)
            end
            # ++
            pkg.depends_on 'python-setuptools' if pkg.install_mode?
        end
        yield(pkg) if block_given?
    end
end

python_package 'external/eigency'

This resolves my issue for the first, but to make sure this works I have to add it to all the package sets with python packages...

This should happen in the python_package definition by default, shouldn't it? Or when calling Rock.activate_python_path?

[doudou]

I'm not familiar with Python packaging, so I can't really answer. If this check is a specific behavior of the eigency package, I would not be very keen on adding it to Autoproj. If it's a general behavior, yes, it should be added to Autoproj.

@2maz ?

[planthaber]

The conflict is that Autoproj does not add non-existent directories to *PATH variables because, you know, they don't exist.
You could create the folder before the installation by adding a post_import block which creates the directory

The overall PYTHONPATH is set by the rock.core package_set here: https://github.com/rock-core/package_set/blob/master/rock/python.rb#L225

So you say this is ineffective in case that folder does not exist?
What do you think about creating the folder in that function to be sure the folder exists and adding the PYTHONPATH is successful as indented there?

[doudou]

The setup function could create the post_import block as I suggested to @hwiedDFKI

But, at the risk of repeating myself:

I'm not familiar with Python packaging, so I can't really answer. If this check is a specific behavior of the eigency package, I don't think it should be generally handled. If it's a general behavior, yes, it should be added generically.

[moooeeeep]

Du to its age, I would assume this issue is no longer relevant today.

If the issue persists, consider to open a new one.