streamlines sdss_install_bootstrap and configuration

[havok2063]

This PR adds a configuration script that runs during sdss_install_bootstrap that prompts the user to define relevant environment variables or accept the default values. It prompts the user the define the following environment variables.

• SDSS_INSTALL_PRODUCT_ROOT
• SDSS_GIT_ROOT
• SDSS_SVN_ROOT
• SDSS_GIT_MODULES
• SDSS_SVN_MODULES
• SDSS_INSTALL_DIR

This configuration setup now allows the user to optionally define a specific product and module path for both git and svn repos, for cases where users already have existing svn and git directories set up and they wish to continue to use these (#12). Otherwise, the original behaviour becomes the default. It also prompts the user if they wish to run module use on their defined module paths, and add these environment variables into their custom sdss_install.yml config file in ~/.sdss_install/sdss_install.yml. sdss_install pulls the environment variables from this location on import and loads them into the os.environ.

I think this should simplify the installation instructions here
down to items a, f, and h modulo some module and GITHUB_KEY stuff. Something like

git clone https://github.com/sdss/sdss_install
cd sdss_install/bin
./sdss_install_bootstrap`

Additional Changes

• Added option to skip installation of git repos inside "version" sub-directories (add option to ignore version subdirectories for git clones #45 )
• Modified the sdss_install bootstrap so it now fresh installs sdss_install
• Adds full test suite.
• Updates ChangeLog for tracking changes in releases
• Synchronizes package version and tag versions (version mismatch between package and tags #43)
• Standard reformatting and linting to improve code readability.

A new option is available now --skip-git-verdirs that does not install git repos into a named-version sub-directory. Installs products using their base product name, a more standard git-thonic way. Also updates the modulefile accordingly.

I've changed the bootstrap installation of sdss_install so it now just installs the latest tag in your product directory, with proper module file. The original checkout used for installation becomes just a temporary checkout. It allows users not to worry about where this gets checked out, or creating directories ahead of time.

I've added some initial tests to make sure sdss_install is working pre/post my changes. Tests ensure stability and robustness of functionality of code during development. All tests should pass. If tests fail after new code is introduced, then new code is breaking something. To run the tests: cd to python/sdss_install/tests and run pytest -v . Travis-CI is now set up properly to automatically run the tests. If the repo indicates "build=failing" it means some tests are broken.

I've updated the package version to be 1.0.7dev in line with the latest tag 1.0.6. bumpversion ensures package versioning can remain in-sync and easily updatable. See here and here. I've also added back entries into the changelog based on the tag descriptions.

[havok2063]

@joelbrownstein @mstalbot I think this PR is ready to go. I've updated the documentation here, https://sdss-install.readthedocs.io/en/config/. See the Intro to sdss_install and options for command-line tool sections. Can you both read through these instructions and try out the installation on your laptops as a new user (i.e. no existing sdss_install module loaded), and make sure the instructions make sense and everything works ok?

The one difference here is after you checkout the temporary sdss_install you will need to cd into it and check out the config branch with git checkout config.

[havok2063]

I've tested this on lore which was a fresh system without sdss_install. The installation worked and I was able to install additional products with it. I'd like to merge this and tag a new version. This shouldn't affect anything at Utah since the default module loaded is tag 1.0.6.

[joelbrownstein]

Once you tag sdss_install that will become the version that is used by the public (bootstrap defaults to the latest tag), so before merging to master (and tagging), we need additional testing to ensure that sdss_install is not suddenly broken for the public.

[havok2063]

That makes sense. Can you define "tests for the public" a bit and clarify what additional tests you want to see? As in what additional systems and set up? So far it's been tested on:

• Mac OS X laptop (Mojave) - with existing SDSS software setup
• a second Mac OS X laptop (High Sierra) - with clean SDSS software setup
• Utah (lore) machine with no prior sdss_install. Clean install of sdss_install, git products and modulefile.
• on Travis-CI which is a fresh AMD 64 Xenial linux box on Python 3.6, 3.7, 3.8. no SDSS setup. https://travis-ci.org/github/sdss/sdss_install

The modifications in this PR only pertain to the installation of sdss_install itself, not of products. Any issues found with product installation would also exist in previous tags. Although I've tested product installation with this branch for various git and svn repos both master/trunk and tags.

We can also only do so much testing. If people find new problems they should submit bug issues so it can be fixed and a new patch can be released. The more tests we add to the test suite here, https://github.com/sdss/sdss_install/tree/config/python/sdss_install/tests, which gets run every time a commit is made, the more robust sdss_install will become.

[joelbrownstein]

It looks like you've tested sdss_install on machines already having TCLSH or LUA modules already installed. But that is the step, I think that has not been fully tested. The issue with sdss_install is the python shell, and because there are many different flavours of modules around, we already know that some of them need a better python shell, which is a solved problem at Utah, but not obvious to someone outside SDSS.

[havok2063]

So can we come up with an itemized list of some combination of operating systems, python versions, and module flavors that you want sdss_install tested on? Having that defined list as a checklist would help to efficiently move forward on implementing improvements for sdss_install. And some fraction of this can be automated with Travis or other continuous integration services.

[joelbrownstein]

That sounds ideal. I would prioritize mac os and linux over windows, and I would prioritize LUA over TCLSH, and perhaps have some documentation in an appropriate location for each combination. Some combinations may ultimately prove that the python shell is (slightly) broken and we could recommend (what usually seems to require) a one-line fix.

For mac, it really boils down to bash which is mostly invariant across mac os versions, so the uncertainty is which modules they install.

For linux, a superuser account (or pc/laptop) is required, so the variations are which linux apt-get or yum-install is used.

[havok2063]

I've added some documentation on testing with the current combinations it's been tested on and some info about the setup Travis is using to test on. See https://sdss-install.readthedocs.io/en/config/test.html. Can you have a look and let me know what you think?

Also, do you think it would be useful to allow users to specify a certain tag of sdss_install to install during the bootstrap step? Otherwise it defaults to the most recent tag, like it does now.