Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Restructuring of folders/links #213

Closed
Tracked by #16
rarkins opened this issue Dec 7, 2021 · 29 comments
Closed
Tracked by #16

Restructuring of folders/links #213

rarkins opened this issue Dec 7, 2021 · 29 comments
Assignees
Labels
breaking Breaking change, requires major version bump priority-3-normal Default priority, "should be done" but isn't prioritised ahead of others status:in-progress Someone is working on implementation type:refactor Refactoring or improving of existing code

Comments

@rarkins
Copy link
Member

rarkins commented Dec 7, 2021

Today we see paths like this:

/usr/local/helm/3.7.1
/usr/local/dotnet
/home/ubuntu/.gem-global/bin
/home/ubuntu/.gem/ruby/3.0.0/bin
/usr/local/ruby/3.0.3/bin
/home/ubuntu/.cargo/bin
/usr/local/rust/1.57.0/bin
/usr/local/go/1.17.4/bin
/go/bin
/usr/local/elixir/1.13.0/bin
/home/ubuntu/.npm-global/bin
/usr/local/yarn/1.22.17/bin
/home/ubuntu/.local/bin
/usr/local/python/3.10.0/bin
/usr/local/poetry/1.1.12/bin
/usr/local/pnpm/6.23.1/bin
/usr/local/php/7.4.26/bin
/usr/local/node/14.18.2/bin
/usr/local/java/11.0.13+8/bin
/usr/local/gradle/6.9.1/bin
/usr/local/composer/2.1.14/bin
/home/ubuntu/bin
/home/ubuntu/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

Can we restructure how we install so that it can be like this?

/home/ubuntu/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

By this I'm hoping what can happen is that every time install-tool installs a version of a too, it symlinks it to /usr/local/bin or /home/ubuntu/bin.

e.g. install-tool npm 8.0.0 might install it into /home/ubuntu/.npm-global/bin but then there's be a symlink from /home/ubuntu/.npm-global/bin/npm to /home/ubuntu/bin/npm.

If we don't have to think about paths and dynamic ENV injection then wouldn't it be much easier to know what's installed?

@viceice
Copy link
Member

viceice commented Dec 7, 2021

Sounds good, but i prefer our shell wrapper and use some fixed files to read appropriate env

@rarkins
Copy link
Member Author

rarkins commented Dec 7, 2021

Are there advantages to having ~20 dirs in the $PATH though? I'm not sure which bits of what I want are conflicting with what you wrote

@viceice
Copy link
Member

viceice commented Dec 7, 2021

i now go the symlink path, should work for root and non-root for node tools.

@rarkins
Copy link
Member Author

rarkins commented Dec 8, 2021

Do you want a separate issue for other tools, so we have something to track?

I think any that we allow runtime install for should be symlinked into the shared bin folders, because otherwise our path will keep growing and growing.

Here's latest renovate/renovate for reference:

/usr/local/dotnet
/home/ubuntu/.gem-global/bin
/home/ubuntu/.gem/ruby/3.0.0/bin
/usr/local/ruby/3.0.3/bin
/home/ubuntu/.cargo/bin
/usr/local/rust/1.57.0/bin
/usr/local/go/1.17.4/bin
/go/bin
/usr/local/elixir/1.13.0/bin
/home/ubuntu/.npm-global/bin
/home/ubuntu/.local/bin
/usr/local/python/3.10.0/bin
/usr/local/gradle/6.9.1/bin
/home/ubuntu/bin
/usr/local/dotnet
/home/ubuntu/.gem-global/bin
/home/ubuntu/.gem/ruby/3.0.0/bin
/usr/local/ruby/3.0.3/bin
/home/ubuntu/.cargo/bin
/usr/local/rust/1.57.0/bin
/usr/local/go/1.17.4/bin
/go/bin
/usr/local/elixir/1.13.0/bin
/home/ubuntu/.npm-global/bin
/home/ubuntu/.local/bin
/usr/local/python/3.10.0/bin
/usr/local/gradle/6.9.1/bin
/home/ubuntu/bin
/home/ubuntu/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

@rarkins
Copy link
Member Author

rarkins commented Dec 8, 2021

Would it be cleaner to put everything in /usr/local/buildpack/ and /home/ubuntu/buildpack/? Then add /usr/local/buildpack/bin and /home/ubuntu/buildpack/bin to PATH

@rarkins
Copy link
Member Author

rarkins commented Dec 8, 2021

jb and composer are the other two tools we support for runtime install today

@viceice
Copy link
Member

viceice commented Dec 8, 2021

mostly done #216

@viceice
Copy link
Member

viceice commented Dec 8, 2021

Would it be cleaner to put everything in /usr/local/buildpack/ and /home/ubuntu/buildpack/? Then add /usr/local/buildpack/bin and /home/ubuntu/buildpack/bin to PATH

Sure, we can do this. will be a breaking, as we need to update ENV PATH=... in Dockerfile

@rarkins
Copy link
Member Author

rarkins commented Dec 8, 2021

No breaking change if we stick to /usr/local/bin and /home/user/bin?

@viceice
Copy link
Member

viceice commented Dec 8, 2021

yes

@rarkins
Copy link
Member Author

rarkins commented Dec 8, 2021

Perhaps not worth it then. Can we nest actual installs inside a buildpack folder though?

@viceice
Copy link
Member

viceice commented Dec 8, 2021

yes, thats possible.

@viceice
Copy link
Member

viceice commented Dec 8, 2021

Can do the refactoring after #218

@viceice
Copy link
Member

viceice commented Dec 9, 2021

current path on full image:

> docker run --rm -it renovate/renovate sh -c 'echo $PATH | tr ":" "\n"'
/usr/local/dotnet
/home/ubuntu/.gem-global/bin
/home/ubuntu/.gem/ruby/3.0.0/bin
/usr/local/ruby/3.0.3/bin
/home/ubuntu/.cargo/bin
/usr/local/rust/1.57.0/bin
/usr/local/go/1.17.4/bin
/go/bin
/usr/local/elixir/1.13.0/bin
/home/ubuntu/.local/bin
/usr/local/python/3.10.1/bin
/home/ubuntu/.npm-global/bin
/usr/local/gradle/6.9.1/bin
/home/ubuntu/bin
/home/ubuntu/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

@viceice
Copy link
Member

viceice commented Jan 21, 2022

We need to move tools below language:

  • /user/local/ruby/3.1.0/bundler/2.3.5
  • /user/local/node/16.0.1/npm/8.3.1

This will allow installing the same version of npm or bundler for different language versions without need to reinstall

@rarkins
Copy link
Member Author

rarkins commented Jan 21, 2022

ok, as long as there's no file name conflicts because that directory name is already used

@viceice viceice self-assigned this Jan 24, 2022
@viceice viceice added priority-3-normal Default priority, "should be done" but isn't prioritised ahead of others status:in-progress Someone is working on implementation type:refactor Refactoring or improving of existing code labels Jan 24, 2022
@viceice
Copy link
Member

viceice commented Jan 27, 2022

We should also make /usr/local/<tool> writable to root group, so we can add tools there as user, so it makes folder and path handling simpler.

Versions installed by root should be protected, so user can't modify them.

We also need to make /usr/local/bin/<tool> executable writable, so user can override that symlink / shell wrapper to a new version too.

We should write current tool version to /usr/local/buildpack/versions/<tool> and /usr/local/buildpack/versions/<lang>.<tool>, eg /usr/local/buildpack/versions/node and /usr/local/buildpack/versions/node.npm.

@viceice
Copy link
Member

viceice commented Jan 27, 2022

I'm open for better paths for version files (#185) and env to an other folder, eg /opt/buildpack/... or /usr/buildpack/... or ... 🤷‍♂️

@viceice viceice added the breaking Breaking change, requires major version bump label Jan 27, 2022
@Chumper
Copy link
Collaborator

Chumper commented Jan 29, 2022

We need to move tools below language:

* `/user/local/ruby/3.1.0/bundler/2.3.5`

* `/user/local/node/16.0.1/npm/8.3.1`

This will allow installing the same version of npm or bundler for different language versions without need to reinstall

Do you have an overview for me why that is needed?
I imagine that we can do this:

install-tool node X
install-tool npm Y

when we need another version of npm we do:

install-tool npm Z

which in the end will just install npm in the needed version.
If it is already on disk, it will just relink or do I miss something?

If we want to do that, then we need to install both at the same time, like
install-tool node X npm Y

or should we install npm generally into the node folder?
So that install-tool npm Y will be installed in the current npm folder .../tools/node/X/npm/Y?

@viceice
Copy link
Member

viceice commented Jan 30, 2022

The last option was my idea. Those where required if we process multiple repo's who require different language version.

Eg first repo needs node v16 and npm V8.
Second repo needs node v14 and npm V8.

Currently we would only install new node version and keep npm. Npm could have resolved peer deps to versions which don't support node v14, so later call can fail.

Not sure if this really happens in npm, but there are other languages and tools where it will happen for sure.

@Chumper
Copy link
Collaborator

Chumper commented Jan 30, 2022

Ok, so we need to make this configurable.
Will see if I can get an example into the branch

@Chumper
Copy link
Collaborator

Chumper commented Jan 31, 2022

@viceice What tools that we currently have should be installed in a subfolder?

Currently I can think of:

  • npm in node
  • yarn in node
  • pnpm in node
  • bundler in ruby
  • poetry in python
  • composer in php (?)
  • maven in java
  • gradle in java
  • scala in java (?)
  • sbt in java (?)

or is this not needed for certain parts?

@rarkins
Copy link
Member Author

rarkins commented Jan 31, 2022

I'm not sure if there's a foolproof way to know when a tool is dependent on the underlying version of the language, or not. Perhaps we need the concept of defining the compatibility, e.g. major/minor/patch. Major compatibility means we'd e.g. want to yarn if the major version of node changed (e.g. 14 -> 16). Minor would mean not reinstalling Poetry if the minor version stayed the same (e.g. 3.8.0 -> 3.8.1). And "patch" can maybe mean "always reinstall" if any language difference (not sure if we have any?).

@Chumper
Copy link
Collaborator

Chumper commented Jan 31, 2022

We can make this as complex as we want, but maybe we have a way to provide a simple solution right now.
@viceice said that there are certain languages and tools where this is definitively required.

My current approach enables each tool to determine on its own if and where it wants to be installed.
So maybe we can start with the absolute basic tools and languages without a concept first and see how we can refactor this when the need arises?

@rarkins
Copy link
Member Author

rarkins commented Jan 31, 2022

Sure, my concern was just if we ended up with hugely bloated runtime images because tools kept being reinstalled every time there's a patch bump of the language

@Chumper
Copy link
Collaborator

Chumper commented Jan 31, 2022

Do you run the images for extended amounts of time?
I imagine that images life for a few hours max. In that time there should a reasonable isntalls of languages and tools only.

The only thing I can imagine is that you run Renovate for a bunch of repos and hey all require different node and npm versions.
In that case we could have n*m installs instead of n+m.

@rarkins
Copy link
Member Author

rarkins commented Jan 31, 2022

Right now the app and WhiteSource Renovate On-Premises essentially run indefinitely, but that might need to change if we bloat the containers too much at runtime

@Chumper
Copy link
Collaborator

Chumper commented Jan 31, 2022

Good to know, in that case we might need to think about how to solve this.

@viceice
Copy link
Member

viceice commented Dec 20, 2022

So the PATH is now smaller again:

>docker run --rm -it renovate/renovate sh -c 'echo $PATH | tr ":" "\n"'                                                              ─╯
/home/ubuntu/.cargo/bin
/home/ubuntu/.local/bin
/go/bin
/home/ubuntu/bin
/opt/buildpack/tools/python/3.11.1/bin
/home/ubuntu/.npm-global/bin
/home/ubuntu/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin

I'm not sure if we can fix python and node global.

I've now implemented a way for ruby to install bundler and cocoapods ruby minor version depended.

eg Bundler v1.17.2 on Ruby v2.6.4: /opt/buildpack/tools/bundler/1.17.2/2.6.0

We probably should close this issue now and open new for the version dependend tools. We can maybe do it like bundler now.

@rarkins rarkins closed this as completed Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
breaking Breaking change, requires major version bump priority-3-normal Default priority, "should be done" but isn't prioritised ahead of others status:in-progress Someone is working on implementation type:refactor Refactoring or improving of existing code
Projects
None yet
Development

No branches or pull requests

3 participants