Make the Erlang installation directory relocatable on Unix-like systems

[JeromeDeBretagne]

Hello,

Here is the second pull request to address ERL-1323 following #2863, this time making the erl and start scripts independent of their actual location. This allows the Erlang installation directory to be moved freely on Unix-like systems.

The idea is to find the absolute path of the script from within itself, as was mentioned in the initial ERL-1323 discussion, thus allowing to set ROOTDIR dynamically when needed. It supports the 2 copies of each script, the ones in erts-%VSN%/bin and the ones in bin.

At the moment, the usage and configuration of FINAL_ROOTDIR is kept as-is for full backwards compatibility with the existing installation scripts. It should be possible to remove FINAL_ROOTDIR totally in a second stage if this new dynamic approach proves to be robust enough on all supported systems, which I expect to be the case.

I've tested the updated erl script a lot and I've applied exactly the same change to the start script but I don't use start myself. So feel free to have a second look at the potential impacts / side effects for start users.

Let me know your thoughts and feedback about this proposition.

Thanks, Jérôme

[kjellwinblad]

I have already merged the first commit in this pull-request to the master branch. Can you remove the first commit and rebase the second commit on master?

[kjellwinblad]

I guess this checks if $newProg is an absolute path but I don't fully understand how it works. Can you explain how expr "x${newProg}" : 'x/' >/dev/null; works?

[JeromeDeBretagne]

Hi Kjell,

Indeed, this checks if newProg starts with a leading / character via a regular expression match:

$ expr "x/test" : 'x/' > /dev/null
$ echo $?
0

$ expr "xtest" : 'x/' > /dev/null
$ echo $?
1

=> the expression returns 0 if there is a match, 1 otherwise.

P.S. The returned output is the number of characters that matched, but this is not relevent hence the >/dev/null part

$ expr "x/test" : 'x/'
2

In the above case, it shows that the first 2 characters of "x/test" have matched with 'x/'

[kjellwinblad]

Thank you for the explanation! The x character confused me.

Is the following:

expr "x${newProg}" : 'x/' > /dev/null"

not equivalent to:

expr "${newProg}" : '/' > /dev/null"

?

[JeromeDeBretagne]

They are not equivalent in all cases depending on the value of newProg. You can see a similar change here:

Be more POSIXly correct in the use of expr.
https://android.googlesource.com/platform/dalvik/+/530e0a50c365bd8451209a014fef4cf293e573c1

that uses exactly the same convention. I won't pretend to be a script guru to explain it in details though.

You can see an example suggesting to use expr X$a = X= reliably here as another reference:
https://pubs.opengroup.org/onlinepubs/009695399/utilities/expr.html

[kjellwinblad]

I see, x is used to make sure "${newProg}" is not interpreted as an operation. I think it would be good to add some comments about the non-obvious parts to make it more readable by people that are not so familiar with shell scripting.

Btw, I have got a question about if one could use a command like realpath instead of doing the link traversal in the script. I will investigate realpath but please let me know if you have some insight into that.

[kjellwinblad]

Thanks for the explanation about realpath.

I think the comments are good. They make it much easier for readers to understand what is happening.

@rickard-green suggested that we could make our own version of realpath (based on the C function realpath) that we can put under the same directory as the erl script. The realpath C function is part of a POSIX standard so it is hopefully available on all relevant platforms. The main benefits of this approach as I see it is that it is potentially more robust and that we would need less complex code in the erl and start scripts. @JeromeDeBretagne What do you think about this idea?

[JeromeDeBretagne]

Since this scripting has been used in the Android Open Source Project for many years, I believe this is a good sign that it is quite rubust. I usually prefer incremental changes, and the current proposal is a limited modification of existing scripts that have proven to support many cases (native, cross, embedeed, etc.) with only an added dependency on ls.

I'm not a fan at all of the idea to add another executable in the Erlang installation directory, needed to simply launch erl, I don't see that as adding robustness. If the Erlang/OTP team is considering bigger changes, I would suggest modifications that vastly simplify the Erlang runtime build and installation, for instance :

• decrease the number of installed executables
• get rid of some scripts in favor of native executables
• remove duplicates between bin/ and erts-$VSN$/bin
• remove the ./Install step
• better support for incrementable builds

That's only my (very personal) view though and it could be discussed within one of the EEF Working Groups I imagine.

@kjellwinblad @rickard-green I still plan to spend more time investigating how to add relevant test cases as suggested, so please let me know if the team still considers this PR a good direction. Thank you.

[kjellwinblad]

@JeromeDeBretagne We have discussed this pull-request a lot internally in the OTP team today. I think you can take a break from working with this until we have decided what approach to take.

There is already a C program call dyn_erl.c that gets installed in bin directory of a release and seems to do similar things as your script. This program was created a long time ago (I did not know about it until today). So other good solutions might be to call that program if $ROOTDIR is incorrect or replace the script with that program. We will let you know when we have made a decision. Thank you for your work so far and sorry for making you do something that might not be accepted.

[JeromeDeBretagne]

Thank you @kjellwinblad for your feedback and no worries at all, I've learnt a lot while working on this contribution.

I didn't know about dyn_erl until today either. It doesn't seem covered much in the official documentation and it's not found in the main bin directory (but it's in the erts-$VSN%/bin). That's what I meant by "simplify" in my previous comment :-)

Good catch, dyn_erl looks very similar to what I was trying to achieve indeed and there is already a "realpath look alike" method commented out. I'm really keen to see which direction the OTP team will take on this topic!

[dgud]

Sorry for jumping in, but @hawk do you remember anything about this?
I guess (without looking) that this is your work a long long time ago,
you had the same ideas and did it for windows, why was it never done for unix?
Any lessons you learned and remember that we don't? :-)

[kjellwinblad]

We use lower_case_with_underscores for variable names in most shell scripts. Can you change the variable names so they use this naming convention?

[kjellwinblad]

It would be great if we could add an automated test for the symlink traversal code. That way we could see if something breaks in any of the machines in Erlang/OTP's testing infrastructure.

A good place to add such a test case would be in this test suite:

$ERL_TOP/erts/test/erlexec_SUITE.erl

You can check if the test is running on a UNIX system similarly to how it is done on line 287 in erlexec_SUITE.erl.

A testcase could check that one can execute different types of symlinks (one or more jumps, relative paths and absolute paths) that points to the erl script.

This document describes how to run test cases:

https://github.com/erlang/otp/blob/master/HOWTO/TESTING.md

Please let me know if you need help with anything related to testing.

[JeromeDeBretagne]

Hello Kjell,

I have already merged the first commit in this pull-request to the master branch. Can you remove the first commit and rebase the second commit on master?

Here is a second revision with the (already merged) first commit removed and the second one rebased on master.

We use lower_case_with_underscores for variable names in most shell scripts. Can you change the variable names so they use this naming convention?

I've switched to the lower_case_with_underscores convention for script variable names.

I think it would be appropriate if you add a comment with a reference to the dx code.

I've also added a comment making a reference to the dx script as suggested.

It would be great if we could add an automated test for the symlink traversal code. That way we could see if something breaks in any of the machines in Erlang/OTP's testing infrastructure.

I'm not familiar at all with the test suite, I'll need much more time to investigate it and see how a test could fit into erlexec_SUITE.erl.

A testcase could check that one can execute different types of symlinks (one or more jumps, relative paths and absolute paths) that points to the erl script.

Please let me know if you need help with anything related to testing.

Do you have a suggestion about how to create the symlinks on the fly as part of one or more tests, especially the ones with an absolute path? Could you share a small sample code maybe?

Thanks, Jérôme

[kjellwinblad]

Hi JeromeDeBretagne,

Do you have a suggestion about how to create the symlinks on the fly as part of
one or more tests, especially the ones with an absolute path? Could you share a small sample code maybe?

You can get the path of a folder where you can put the new symlinks by executing this:

filename:join(filename:dirname(code:which(?MODULE)), "erlexec_SUITE_data")

There is an Erlang function for creating symlinks.

The filename:absname functions are convenient for getting an absolute path from a relative path.

We use the common test framework for our test suits.

The os:cmd/1 function can be used to execute external commands.

It can be useful to know that you can build and execute a single test case in a test suite by running a single command. For example, to execute the args_file test case in erlexec_SUITE (make sure that ERL_TOP is set to the root of Erlang/OTP source tree):

(cd $ERL_TOP/erts/ &&  make ARGS="-suite erlexec_SUITE -case args_file" test)

Please let me know if you get stuck on something and I will try to help you.

[JeromeDeBretagne]

Hi @kjellwinblad ,

Thanks for the inputs, I'll do my best to look into this in the coming days.

I'm currently spending some time back on #2863 as Lukas has detected that it could be the source of test case regression. Your second look would be appreciated on that one too.

[JeromeDeBretagne]

Hi @kjellwinblad , Hi @dgud,

We will let you know when we have made a decision. Thank you for your work so far and sorry for making you do something that might not be accepted.

As OTP 24 is getting closer, do you have a recommandation about the approach the team would prefer on this topic (#4461)?

Could this first PR be accepatble for OTP 24, with an improvement to investigate for OTP 24.X or 25 later on?

If the OTP team would prefer a bigger/better change in one go, could you drop some hints about a more acceptable direction, so that the community can start investigating it and try to propose another PR?

Thank you, Jérôme

[hawk]

I did this 5 years ago for escript. Regardless of the final solution, I think it is good if the same solution can be used for all ways of starting the system.

1d8dbc7

[kjellwinblad]

Hi @JeromeDeBretagne,

Sorry for the long time to respond. I will discuss with the team if it is possible to bring this PR into 24. Unfortunately, I think the most likely answer is that it will be too late (which is our fault) as we have already released rc2.

If the OTP team would prefer a bigger/better change in one go, could you drop some hints about a more acceptable direction so that the community can start investigating it and try to propose another PR?

Yes, we have a ticket in our backlog about deciding what direction we want to go. We will update this PR when we have made a decision.

@hawk thanks. Yes, it would be good to reuse the same solution in all ways one can start the system.

[JeromeDeBretagne]

Hi @kjellwinblad,

Thanks for the update, waiting for the team feedback then!

[kjellwinblad]

@JeromeDeBretagne wrote:

Thanks for the update, waiting for the team feedback then!

Now, we have finally decided on which direction we want to go with this. Thanks for your patience.

We have decided and discussed the following:

• Having the installation directory relocatable is something we want to have ready for the Erlang/OTP 25 release. Preferable as soon as possible so we can test it well before the release.
• We want to have a single way to get the absolute path of the installation directory. We, therefore, want to replace the erl script with a C program. The dyn_erl.c program seems to do what we want but we have to make sure that nothing breaks when replacing the erl script with this program.
• We need to investigate if the start script needs to be kept as a script (it might be documented that users can modify the start script to customize the start). If start needs to be kept as a script, we can get the absolute path of the installation by calling a C program (preferable the same as the one that will be the new erl program). The program that we call from the start script must always be at the same location relative to the start script.
• The install script also adds absolute paths to the RELEASES file. The RELEASES file is used when doing upgrades. We have to replace the absolute paths in the RELEASES file with relative paths and ensure that this does not break anything. There are some test cases in the SASL test suite related to release handling that one can use to test this, but it is not certain that these tests cover everything.
• We should also try to make sure that the changes we do to make an Erlang installation relocatable do not break other tools. @ferd and @tsloughter do you think the change described in this message will affect rebar3 in any way?

@JeromeDeBretagne, if you want to continue to work on this, we would be happy to assist you in whatever way we can.

[tsloughter]

I don't think this will effect rebar3. Currently releases generated with rebar3 will create a RELEASES file that uses relative paths, like {sasl,"4.0.1","./lib/sasl-4.0.1"},

This will be great, I was recently dealing with adding support for installing binary bundles of OTP to erlup (https://github.com/tsloughter/erlup/) and had forgotten about this issue until realizing I had to run INSTALL from the bundle because of paths needing to be setup in multiple files.

[JeromeDeBretagne]

Thank you for sharing the goals and steps around this evolution, that sounds like a good plan. I can't tell yet if I'll have the bandwidth to look at it soon, I'm working on another evolution atm.

It seems it will require slightly different skills compared to my original PR (which was simply reusing existing scripts) with some changes to the build process needed this time. That's not the easiest part of the Erlang project to modify, at least for me.

To raise awareness with the community, I would suggest to create a dedicated issue with the help-wanted flag (or a few more granular ones?) replacing #4461 and describing the target changes. Then let's see how to get the discussion started there, with one or more contributions either from the OTP team or the community. What do you think?

In fact, if you already have a list of goals for the OTP 25 release that you can share publicly, it would be great if you could describe them in a similar way through the issue tracker, to see if some of them could be worked on by the community. Just my 2 cents.

[kjellwinblad]

@JeromeDeBretagne Thanks for your suggestion in #2879 (comment) . I think your suggestions are good and have discussed them with the Erlang/OTP team at Ericsson. To formalize goals and planned features in a way that is accessible to the community requires some work so there is a trade-off that needs to be made. That being said, we are always trying to improve our way of working so it is very possible that we will try to become more public about goals so that we can get more help from external contributors.

FYI, I'm starting to work on implementing what is outlined in #2879 (comment) now.

[ferd]

I don't think this will effect rebar3. Currently releases generated with rebar3 will create a RELEASES file that uses relative paths, like {sasl,"4.0.1","./lib/sasl-4.0.1"},

This will be great, I was recently dealing with adding support for installing binary bundles of OTP to erlup (https://github.com/tsloughter/erlup/) and had forgotten about this issue until realizing I had to run INSTALL from the bundle because of paths needing to be setup in multiple files.

The one edge case I recall hitting with releases was specifically with relups. IIRC the issue is that Rebar3 uses the relative paths (a good thing for relocatable stuff) while the VM uses absolute paths, which had two unforeseen consequences before this PR:

1. The release boots fine, and upgrades fine with a relup, but will not rollback to the first version because of the path difference. Further installs and unpacks worked fine
2. If the code in the release changes directories from the default one at startup (using c:cd/1 or file:set_cwd/1), then the operations around SASL and the release handlers started failing.

Being able to have the release know about the original absolute release path and use the RELEASES file's relative paths together would likely improve things on both issues.

[kjellwinblad]

I have created another PR (#5427) implementing the suggestion outlined in #2879 (comment). We can close this PR if that PR gets merged.

@ferd Thank you for your feedback. I have added a test case that (I think) tests the two unforeseen consequences you described, and it seems to be working fine now. Btw, your book was of great help to me when trying to figure out how release handling in Erlang works.

[garazdawi]

#5427 is now merged, so I'm closing this PR. Thank you @JeromeDeBretagne for your help and for pushing us to finally fix this problem.