Env inheritance for RUBYLIB

[2maz]

@doudou I am trying to allow an Debian packaged 'orogen' to be used by autoproj.
Therefore I export an updated RUBYLIB by source a package specific env.sh.
While run 'orogen' directly in a shell without a problem, using it through autoproj fails - since RUBYLIB env var does not contain the entries of the original env anymore:

/opt/rock/master-20.06/rock-master-20.06-ruby-orogen/bin/orogen:4:in
`require': cannot load such file -- orogen (LoadError)
         from
/opt/rock/master-20.06/rock-master-20.06-ruby-orogen/bin/orogen:4:in
`<main>' 

Autoproj.isolate_environment is disabled, so I would expect the environmental setting of RUBYLIB to be available and honoured by autoproj.
It looks to me as if there is some special treatment regarding the RUBYLIB env var, so hints on how to properly handle this situation are more than welcome.

[doudou]

Tbh, I'm not really sure. Maybe a bug, yes. The ability to inherit environment has always been rather fragile - it's not simple doing it right, and in the case of Ruby we get through the loops of resolving the environment RUBYLIB using Bundler as well. Now, if there is a bug, it probably exists ever since 2.0 was out.

Now, in addition to that, I'm rather keen on having isolate_environment working. It really really should. Forcing people to not set isolate_environment is not a very nice idea...

How do you handle other ruby-based packages ? Or is that the first one ?

I would suggest shipping an init.rb that sets up the environment. Something like a scheme where each package can have a share/autoproj/<package_name>.rb and an init.rb within the package set that loads these files ? Then ship one for orogen ?

[2maz]

Tbh, I'm not really sure. Maybe a bug, yes. The ability to inherit environment has always been rather fragile - it's not simple doing it right, and in the case of Ruby we get through the loops of resolving the environment RUBYLIB using Bundler as well. Now, if there is a bug, it probably exists ever since 2.0 was out.

Looks like the reason is resetting the RUBYLIB to nil in .autoproj/bin/amake

# Autoproj generated preamble, v1
if defined?(Bundler)
    if Bundler.respond_to?(:with_unbundled_env)
        Bundler.with_unbundled_env do
            exec(Hash['RUBYLIB' => nil], $0, *ARGV)
        end
    else
        Bundler.with_clean_env do
            exec(Hash['RUBYLIB' => nil], $0, *ARGV)
        end
    end
elsif ENV['RUBYLIB']
    exec(Hash['RUBYLIB' => nil], $0, *ARGV)
end

I would suggest shipping an init.rb that sets up the environment. Something like a scheme where each package can have a share/autoproj/<package_name>.rb and an init.rb within the package set that loads these files ? Then ship one for orogen ?

I prefer not to have anything autoproj specific in the Debian packages, but rather leave it with providing an env.sh per package.
But I now allow to maintain an isolated environment, by evaluating the Debian packages' env.sh in isolation files and updating autoproj via env.add_path accordingly.

Could also be part of autoproj (maybe differently implemented)?: any script that is embedded with Autobuild.env_source_file would be evaluated in isolation and the resulting env applied internally.
My initial issue is only encountered because Autobuild.env_source_file only affects the generated env.sh, but not autoproj's internally used env.

[doudou]

But I now allow to maintain an isolated environment, by evaluating the Debian packages' env.sh in isolation files and updating autoproj via env.add_path accordingly.

This is a textbook example of "let's try to guess what we should be told". At the very least, export on build a yaml or json file that describes the environment along with the env.sh and load that.

[2maz]

This is a textbook example of "let's try to guess what we should be told". At the very least, export on build a yaml or json file that describes the environment along with the env.sh and load that.

Happens since autoproj does not support just loading a custom environment setup script !? For instance the configured result of https://github.com/orocos-toolchain/rtt/blob/master/env-hooks/00.rtt.sh.in.
Though I am fine with adding a generated yaml or json, this does not provide a solution - at least I do not see it.

[doudou]

Happens since autoproj does not support just loading a custom environment setup script !?

If we support loading an env.json or env.yml that does not require to assume a specific format you provide a mechanism that's not specific to autoproj and is a lot more robust, rather than a file that is a full programming language and is written by a human.

You don't want to add an autoproj-specific init.rb in the packages. I don't want to have to load a full shell script to guess the environment. What I propose is called a middle-ground.

[2maz]

Please note, that this issue points one thing out: the isolated environment for calls from within autoproj is different from the one when sourcing autoproj's env.sh in a clean environment.

If we support loading an env.json or env.yml that does not require to assume a specific format you provide a mechanism that's not specific to autoproj and is a lot more robust, rather than a file that is a full programming language and is written by a human.

That is and has been clear to me, and I can surely generate env.json/yml for the ones that 'apaka' is generating based on autoproj's available information.
But, and I am sorry to repeat myself, what is your actual suggestion or answer to handling third party env.sh files - I am mean not the ones apaka is rendering from autoproj's package definitions?
Though experiment: how would you generate an env.json/yml instead of an env.sh in autoproj?

[doudou]

Please note, that this issue points one thing out: the isolated environment for calls from within autoproj is different from the one when sourcing autoproj's env.sh in a clean environment.

So far what I got was that the non-isolated environment was different since I thought you were trying to get Autoproj to pick up an environment you set externally (which is indeed a bug, only one that I would put all the way down in the pile of bugs to fix). The env.sh after a call to isolated_environment should reset e.g. RUBYLIB.

But, and I am sorry to repeat myself, what is your actual suggestion or answer to handling third party env.sh files - I am mean not the ones apaka is rendering from autoproj's package definitions?

I don't, I consider this a bad idea. I duplicate in autoproj the logic that the package-generated env.sh has, but this time actually having the information. Until now (10 years of autoproj), I have yet to find a package where this is really problematic.

For instance, the RTT-generated env.sh you pointed out is guessing the relationship between CMAKE_PREFIX_PATH and RTT_COMPONENT_PATH (and, of course, assumes the CMAKE_PREFIX_PATH is fully set before the RTT_COMPONENT_PATH is generated). The autoproj packages each know what to put in RTT_COMPONENT_PATH.

Same for OROCOS_TARGET. The Autoproj package sets should ensures that everyone is using the same OROCOS_TARGET by passing it to the relevant CMake variables or whatever other means are used.

[2maz]

FYI: starting with the current Rock Debian Package Release (master-20.06) an env.yml is provided per package, and already used to update the env.sh in rock-osdeps-package_set