[Tools] Ubuntu production environment "bootstrap" script

[johnsaigle]

This pull request has been heavily refactored so a lot of the discussion may not make sense.

Brief summary of changes

bootstrap verifies loris dependencies such as Apache/PHP versions and misc system tools and optionally installs them. This is meant to be run only on development environments as production environments will in the future have all of this handled by a packaged release.

PHP_CLI_Helper consists of helper functions to make it easier to write PHP scripts for the command line.

Further work

• Make something equivalent for CentOS as it's another environment that our team uses for development.

To test this change...

• tools/bootstrap and let me know if there are any errors!

[johnsaigle]

This functionality is preserved by setup and bootstrap; one can add a 'dev' argument upon invoking these scripts and have composer do the appropriate thing.

[PapillonMcGill]

since other lines end with a dot, this line should also end with a dot
same of other doc block

[driusan]

Replacing a simple < 10 line Makefile with hundreds of lines of bespoke PHP scripts in order to adopt a third party's internal procedures is not a step forward.

If you want to simplify running the setup add some standard targets like make clean, make test or make install.

[ridz1208]

hmmm, let's see:

• first, the <10 line makefile is doing soooo much less than the hundreds of lines of bespoke php
• make is a tool not a standard and we are in dying need of standards (especially some not defined by LORIS for LORIS) and these ones have some documentation and play ball with github... especially that they are described by GITHUB Eng and we use github for PRs, issues, projects, why not standards !!
• Talking about standards, the intent of scripts to rule them all is to standardize these scripts in multi-project development and seems to me that LORIS, LORIS-MRI, CCNA, IBIS, BIOBANK, etc... do fit that picture pretty nicely. imagine having a standard way of setting them up or cheking for dependencies istead of ad hoc-ing our way during the installation.
• let's also not omit the fact that we can write these scripts in whatever language we want and we can have them do everything we need. no need to code if and for loops in bash !!! no we can use python, php what ever... that sounds pretty nice with an actual potential of being maintained
• and how about just this... someone actually made the effort to code this and test it instead of just talking about it and dreaming.

the way I see it, if LORIS is WAY TOO ATTACHED to makefiles, then just create makefiles and have them called FROM the scripts to rule them all, i dont care as long as I dont have to explain to each and every new developer how to run a test, how to run phpcs, how to run phan, ...

a quick read https://www.jefframnani.com/project-build-protocol/

[driusan]

I'm just going to copy and paste the blog post that you linked to here which applies to this repo:

Scenario One: One Project, Many Commands

In this scenario, you’re mostly working on one main app all day. Your company’s monolith, for example.

Let’s say you’re working in a Django project. To run the Django unit tests you run:

DJANGO_SETTINGS_MODULE=myapp.settings.local_test python manage.py test

To run the JavaScript front-end tests you run:

node_modules/.bin/gulp test

Now, I’m lazy. I don’t want to have to remember these two commands, their environment variables, or their arguments. What I normally do in this situation is write a simple Makefile.

test: test-py test-js

test-py:
    DJANGO_SETTINGS_MODULE=myapp.settings.local_test REUSE_DB=1 ./manage.py test

test-js:
    node_modules/.bin/gulp test

Now when I come back to this project, I only have to remember to run make
test, and all my tests get run.

(In my real Makefile I use .PHONY targets to tell Make that these names don’t correspond to actual files.)

For this example, I only created a test target, but the common targets I like to use are: build, test, run, and dist. These targets create a common protocol I can depend on. They also serve as a form of executable documentation for how to do one of these tasks.

Makefiles can get pretty hairy, pretty quickly, so the minute a task starts getting hard to read or maintain, I’ll write a separate script to perform the task, then just delegate to the external script within the Makefile.

Why Make? Because it’s ubiquitous and it’s easy to install.

At 9 lines which don't branch or loop, you can't even claim we're at the "pretty hair, pretty quickly" point that would necessitate delegating to an external script inside of the Makefile. The complexity in this PR is added by removing the Makefile and attempting to rewrite the functionality in another language that isn't well suited to the task of building software.

The other scenerio in the blog post (which is a little tortured to say applies to LORIS, although arguably does with Loris-MRI) doesn't provide any rational for abandoning that approach rather than just standardizing the makefile targets across the projects. Here are a list of standard Makefile targets used by GNU: https://www.gnu.org/prep/standards/html_node/Standard-Targets.html (not all apply to LORIS. I'm also surprised that "test" isn't included.) Adopting a real standard that's been standardized for building software for decades rather than a blog post standard would have a number of benefits such as making it trivial to build packages for any supported OS using their standard procedures.

[ridz1208]

@driusan

Call me cynic but I think

Makefiles can get pretty hairy, pretty quickly, so the minute a task starts getting hard to read or maintain, I’ll write a separate script to perform the task, then just delegate to the external script within the Makefile.

applies to us rather well.

I think that even though it is using another language that isn't well suited to the task of building software it might have more chances to be maintained by your average LORIS developer who probably hasn't touched a make file in the last 10 years.

At 9 lines which don't branch or loop, you can't even claim we're at the "pretty hair, pretty quickly" well that's because the 9 lines are not doing much. I'd be curious to see what the makefiles would look like with the same functionality as the scripts to rule them all (checking for versions, maybe handling errors....)

The other sceario in the blog post (which is a little tortured to say applies to LORIS, although arguably does with Loris-MRI) in our effort to streamline project releases, I'm surprised you don't see the potential of using this for projects, I would loove to havea single lioner for every release scripts/update. And wait... I thought the plan was to extract all modules into individual repos... that sounds pretty hairy to me.

As far as the standards are concerned, you wanna choose the make standards over the scripts to rule them all, by all means do, but no one has taken initiative to do that and I don't see anyone doing it... (test is included in the make check btw) like I said earlier we gonna appreciate the fact that the work has been done !!! I'm not saying to merge this blindly but it's at least worth a discussion and some testing.

I would like to see a comparison between the GNU way that's been used for "decades" vs the github way that might be used in the next decades ?? who knows maybe we'd actually be on to something futuristic there

[johnsaigle]

Just to be clear this PR wasn't written for the purpose of replacing the Makefile. I did the development toward creating these scripts and realized that they covered what the Makefile did (and a bunch more, obviously). I'm not opposed to having both live side-by-side. It's no trouble to add the file back and do something like below if we want Makefiles.

test:
    script/test

I do believe that the bootstrap and test logic is pretty complex/"hairy" to do in Bash. Without getting in to language features, more developers on our team are skilled with PHP over Bash and so I wrote these so as to be more easily maintainable for a greater number of people. (Same as the update script).

[johnsaigle]

TODO:

• Revert changes to Makefile and travis.yml
• Rename instances of scripts to tools
• Document changes, etc. "tools/bootstrap.php can be used to verify or install missing system requirements for Ubuntu. Used only for dev environments...."
• Delete setup in favour of make clean target with the same bash commands
• Add a make test target that runs tools/test.php. This script may need to be scrapped; will coordinate with @driusan and PR Add make check target #4002

Pretty sure this is resolved

[PapillonMcGill]

almost there!

[driusan]

Other than all the references to "bash", this isn't conforming to the discussion we had last year about how this should be approached. This is still trying to install everything, when what we need is a script which verifies dependencies and reports the missing/incorrect versions of things that are installed before any attempt is made to install anything.

The PR itself should also be closed and re-sent when it's ready so that the discussion doesn't get confused with previous discussions from before major refactoring.

[driusan]

This isn't using bash, it's using the shell

[driusan]

I think this would be more reliable if it used exec and checked the shell return code, rather than trying to parse stdout.

[driusan]

Are you sure about that? dpkg(1) says:

EXIT STATUS         top

       0      The requested action was successfully performed.  Or a check
              or assertion command returned true.

       1      A check or assertion command returned false.

       2      Fatal or unrecoverable error due to invalid command-line
              usage, or interactions with the system, such as accesses to
              the database, memory allocations, etc.

(but doesn't really clarify what "a check or assertion command" is)

[johnsaigle]

Hmm well I mean the -s flag gives the status of a package and returns 1 when not found so I think it works in this case.

[driusan]

The second sentence is unnecessary.

[driusan]

What if PHP isn't installed?

[johnsaigle]

I'll add a check for it

[driusan]

I mean the script is written in PHP, so how can it check that?

[johnsaigle]

Closing this PR and will re-open a fresh one.