double code & dysfunctional master branch when downloading package

[filchristou]

when add ReinforcementLearning is executed I have double code in my machine:
#---------- ~/.julia/packages/ ----------#
./ReinforcementLearning//ReinforcementLearningBase/
./ReinforcementLearning//ReinforcementLearningCore/
./ReinforcementLearning//ReinforcementLearningEnvironments/
./ReinforcementLearning//ReinforcementLearningZoo/
./ReinforcementLearningBase/
./ReinforcementLearningCore/
./ReinforcementLearningEnvironments/
./ReinforcementLearningZoo/

why is this needed ?

Problem
when add ReinforcementLearning#master is executed the Base,Core,Environments,Zoo packages are not updated:
./ReinforcementLearning//ReinforcementLearningBase/ --> updated (not used)
./ReinforcementLearning//ReinforcementLearningCore/ --> updated (not used)
./ReinforcementLearning//ReinforcementLearningEnvironments/ --> updated (not used)
./ReinforcementLearning//ReinforcementLearningZoo/ --> updated (not used)
./ReinforcementLearningBase/ --> old (the one used)
./ReinforcementLearningCore/ --> old (the one used)
./ReinforcementLearningEnvironments/ --> old (the one used)
./ReinforcementLearningZoo/ --> old (the one used)

And since locally the base packages are used, this basically means that I cannot test the master branch.
I guess this is not the desired behavior..
so how to work with the master branch ?

problem also explained here: https://discourse.julialang.org/t/cannot-run-reinforcementlearning-master/64050

[findmyway]

Hey @filchristou ,

Thanks for raising this problem.

First, have you read https://juliareinforcementlearning.org/docs/tips/#How-to-setup-local-development-environment?

In the latest stable release (and the master branch), we only have one folder. So there should only be one ReinforcementLearning in your ~/.julia/packages/. But in some cases, you might have installed an old version of ReinforcementLearning.jl, then you'll see those other subpackages.

when add ReinforcementLearning#master is executed the Base,Core,Environments,Zoo packages are not updated:

You might try ]dev ReinforcementLearning instead based on the doc link above.

Let me know if you still have any questions.

[filchristou]

Hi. indeed ]dev ReinforcementLearning downloads only one whole ReinforcementLearning package and now I can test it. Thanks.

First, have you read https://juliareinforcementlearning.org/docs/tips/#How-to-setup-local-development-environment?

Okey. Now I see that this package had this double code peculiarity, which now should not be the case:

In the latest stable release (and the master branch), we only have one folder

Still whenever I ]add ReinforcementLearning or even ]add ReinforcementLearning#master all packages ./ReinforcementLearning{Base,Core,Enviroments,Zoo} are also downloaded. I also made sure to have a clean system (]rm ReinforcementLearning plus manually deleting ~/.julia/packages/ReinforcementLearning* plus ]gc).
Moreover these packages are not in accordance when I download the master branch (code in ./ReinforcementLearning is updated but in ./ReinforcementLearning{Base,Core,Enviroments,Zoo} not ).

So if I understand correctly that the desired behavior should be only one folder in ~/.julia/packages/, then this is not the case for me.

[findmyway]

Still whenever I ]add ReinforcementLearning or even ]add ReinforcementLearning#master all packages ./ReinforcementLearning{Base,Core,Enviroments,Zoo} are also downloaded. I also made sure to have a clean system (]rm ReinforcementLearning plus manually deleting ~/.julia/packages/ReinforcementLearning* plus ]gc).
Moreover these packages are not in accordance when I download the master branch (code in ./ReinforcementLearning is updated but in ./ReinforcementLearning{Base,Core,Enviroments,Zoo} not ).

OK, I just did a quick test in docker and I can confirm it. I think we hit some corner cases in the implementation of Pkg.jl. Let me take a deep look into the related source code in Pkg.jl first. If I can't figure out the reason, I'll rephrase this problem and create an issue there. Thanks again!

[findmyway]

It seems the dev and add commands adopt different strategies to handle the subdir packages.

For example, if I ]add ReinforcementLearningBase, the ReinforcementLearningBase will be created in ~/.julia/packages. But ]dev ReinforcementLearningBase will create the whole ReinforcementLearning package under ~/.julia/dev. The design decision behind is unclear to me.

[findmyway]

Quote reply from Kristoffer Carlsson on slack:

While it could be possible for dev to only give you that subdir path in dev it hasn't really been implemented since the first thing many people do after dev a package is to start making commits

[filchristou]

but it seems that this still doesn't explain the behavior of double code in case of ]add ReinforcementLearning.
Can you please post here the solution/fix or the possible next issue url ?
I would like to follow the problem down the road to get a better insight.

[findmyway]

but it seems that this still doesn't explain the behavior of double code in case of ]add ReinforcementLearning

That's because ReinforcementLearning depends on ReinforcementLearningBase/Core..., similar to other dependents, they appear under the ~/.julia/packages too. I guess Pkg.jl currently only looks for packages under that dir, not recursively.

[filchristou]

but again. this doesn't explain why when I download the #master branch, the code is different between the main directory and the dependent-directories..

Moreover these packages are not in accordance when I download the master branch (code in ./ReinforcementLearning is updated but in ./ReinforcementLearning{Base,Core,Enviroments,Zoo} not ).

[findmyway]

but again. this doesn't explain why when I download the #master branch, the code is different between the main directory and the dependent-directories..

Moreover these packages are not in accordance when I download the master branch (code in ./ReinforcementLearning is updated but in ./ReinforcementLearning{Base,Core,Enviroments,Zoo} not ).

When you run ]add AnyPackage#master, its dependents are not updated. And ReinforcementLearning.jl is nothing special here. So you need to manually run ]add ReinforcementLearningBase#master... to get the master branch of its independents. However, with ]dev ReinforcementLearning, you don't need to ]dev ReinforcementLearningBase..., that's because subpackages are contained in the same repo and enter the develop mode by default.

[filchristou]

Okey. Now it's clear. thanks a lot.