Add config.yaml file #2152

tgamblin · 2016-10-27T19:41:17Z

Rework of #908. Fixes #635, #1925.

This is a continuation of @hegner's #908, which allowed customizing the install location. This adds a top-level config.yaml with many commonly set properties. config.yaml consolidates a lot of the things currently in spack/__init__.py.

Todo:

Below is what the config.yaml file looks like currently. Just need to wire in the features.

Thoughts? @alalazo @adamjstewart @mamelara @mathstuf @justintoo @hegner

config:
  # This is the path to the root of the Spack install tree.
  # You can use $spack here to refer to the root of the spack instance.
  install_tree: $spack/opt/spack


  # Temporary locations Spack can try to use for builds.
  #
  # Spack will use the first one it finds that exists and is writable.
  # You can use $tempdir to refer to the system default temp directory
  # (as returned by tempfile.gettempdir()).
  #
  # A value of $local indicates that Spack should run builds directly
  # inside its install directory without staging them in temporary space.
  build_stage:
    - /usr/workspace/*/%u
    - $tempdir
    - /nfs/tmp2/%u
    - $local


  # Cache directory already downloaded source tarballs and archived
  # repositories. This can be purged with spack purge
  source_cache: $spack/var/spack/cache


  # Cache directory for miscellaneous files, like the package index.
  misc_cache: ~/.spack/cache


  # If this is false, tools like curl that use SSL will not verify
  # certifiates. (e.g., curl will use use the -k option)
  verify_ssl: true


  # If set to true, Spack will always check checksums after downloading
  # archives. If false, Spack skips the checksum step.
  checksum: true


  # If set to false, `spack install` and friends will NOT clean
  # potentially harmful variables from the build environment. Use wisely.
  clean: true

adamjstewart · 2016-10-27T19:46:47Z

This looks awesome!

Only thing, can we change:

dirty: false

to

clean: true

@citibeth has mentioned avoiding negative logic before and I think it makes sense in this case.

hegner · 2016-10-27T19:50:13Z

Looks very good! Maybe docs could be added to your checklist.

tgamblin · 2016-10-27T20:00:01Z

@adamjstewart: is dirty: false actually negative logic? dirty is the special case and we're saying it's true or false... not_dirty: true would be negative logic here I think.

I don't feel too strongly about this but dirty: true corresponds to the parameters on the commands.

adamjstewart · 2016-10-27T20:02:51Z

@tgamblin In this case, "dirty" implies "not clean". It's a matter of whether or not we are cleaning up environment variables, not a matter of whether or not we want Spack to be "dirty". But I understand it matches the option flag. I guess the annoying thing is we don't have an option with the flag name.

tgamblin · 2016-10-27T20:03:18Z

I could just was easily argue that clean implies not dirty...

adamjstewart · 2016-10-27T20:13:53Z

Then why did you use "NOT clean" in the description of the flag? Because "dirty" doesn't actually mean anything 😆

citibeth · 2016-10-27T21:05:11Z

I agree it's ambiguous here which one is negative logic. But I think the
"clean" terminology has the edge here because "dirty" has bad
connotations. Other things to consider might be "keep environment", which
is a more neutral term for what dirty does.

On Thu, Oct 27, 2016 at 4:13 PM, Adam J. Stewart notifications@github.com
wrote:

Then why did you use "NOT clean" in the description of the flag? Because
"dirty" doesn't actually mean anything 😆

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2152 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cdzFlVJqWaPSEL_tbugktyGSDDTOyks5q4QYDgaJpZM4KivqN
.

tgamblin · 2016-10-28T08:01:31Z

Ok, everything but docs is done. Can people try this out?

I went with clean instead of dirty in config.yaml, and added --clean args to spack setup, spack diy, and spack install so you can be clean when Spack is dirty and vice versa.

alalazo · 2016-10-28T08:48:51Z

From the tests :

==> List of failing tests:
  test_chdir (spack.test.stage.StageTest)
  test_expand_archive (spack.test.stage.StageTest)
  test_expand_archive_with_chdir (spack.test.stage.StageTest)
  test_fetch (spack.test.stage.StageTest)

citibeth · 2016-10-28T12:05:49Z

Now that we have a config.yaml and a re-work of dirty/clean... would it be possible to integrate the functionality I removed in #1818 into config.yaml? (That is... add an option to config.yaml that is equivalent to always saying spack --dirty)

tgamblin · 2016-10-28T14:54:59Z

@citibeth: I'm confused. That's is exactly what the "clean" option is. Note this also fixes #1925.

Maybe we should call it dirty after all?

adamjstewart · 2016-10-28T15:06:09Z

That is... add an option to config.yaml that is equivalent to always saying spack --dirty

@citibeth That's what this PR does. You can set clean: False in config.yaml so Spack never cleans your paths. This can also be overridden on a case-by-case basis by running spack install --clean if you need to.

citibeth · 2016-10-28T15:21:00Z

Great, thanks!

On Fri, Oct 28, 2016 at 11:06 AM, Adam J. Stewart notifications@github.com
wrote:

That is... add an option to config.yaml that is equivalent to always
saying spack --dirty

@citibeth https://github.com/citibeth That's what this PR does. You can
set clean: False in config.yaml so Spack never cleans your paths. This
can also be overridden on a case-by-case basis by running spack install
--clean if you need to.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2152 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cdyNV41iw6gPqSs9gTWH9zvx1VLTCks5q4g9kgaJpZM4KivqN
.

pramodk · 2016-10-28T18:08:28Z

Wouldn't it be useful to have an option for --log-format as well?

tgamblin · 2016-10-28T18:23:00Z

@pramodk: what do you mean? You want to be able to default the log format to something? I think the raw output is a good default, and the XML output is good for tests -- is there a reason you'd want the XML output all the time?

pramodk · 2016-10-28T18:40:18Z

I mean to enable --log-format. Correct me if I have misunderstood: XML report for a package is generated in var/spack/junit-report only when I specify --log-format=junit. I was wondering about an option to enable this (which could be useful during testing (?)).

tgamblin · 2016-10-28T18:46:05Z

That might be useful -- let's get this merged and then see if we need it. I was thinking that for testing we would likely just call the command with the --log-format=junit option in the test script.

tgamblin · 2016-10-28T23:18:58Z

Ok this is all done and should be ready to go. Bugs are fixed and docs are in. Can I get a review?

@alalazo @adamjstewart @mamelara @mathstuf @justintoo @hegner

adamjstewart · 2016-10-29T01:05:10Z

lib/spack/spack/config.py

-Will make Spack take compilers *only* from the user configuration, and
-the site configuration will be ignored.
+See the documentation on :ref:`configurations scopes
+<configuration-scopes>`_ for details on how Spack's configuration system


The trailing underscore is causing the documentation tests to fail. I think the trailing underscore is only needed for external URLs, not internal references. Gotta love .rst!

tgamblin · 2016-10-29T02:56:41Z

@adamjstewart Ok fixed up the docs. Added a bit more text to the page for spack.config, and the page for spack.schema now shows the schemas in the docs.

tgamblin · 2016-10-29T03:00:42Z

Also, @hegner: I took out the ability to customize the DirectoryLayout class used for the install prefix.

My rationale is that I think if we allow install tree layouts to be customizable, it should be customizable through some configuration (e.g., install_tree.yaml?) in the directory layout itself, so that switching install trees does not require changing two configuration options (which is error-prone). Do you need that feature or can that wait?

citibeth · 2016-10-29T15:00:44Z

etc/spack/defaults/config.yaml

+#
+# Per-spack-instance settings (overrides defaults):
+#   $SPACK_ROOT/etc/spack/config.yaml
+#


etc/spack/config.yaml should be added to .gitignore. We don't want to normally check site configuration back into Spack. (If this is not added, config.yaml will end up accidentally in some PRs).

Yes. /etc/spack/*.yaml is already in .gitignore

citibeth · 2016-10-29T15:01:20Z

etc/spack/defaults/config.yaml

+  # This is the path to the root of the Spack install tree.
+  # You can use $spack here to refer to the root of the spack instance.
+  install_tree: $spack/opt/spack
+


Consider making $spack upper case, to align with common bash usage.

I really hate uppercase. And this is YAML. Also it's been this way for a while... the default repo location uses this syntax in /etc/spack/defaults/repos.yaml

citibeth · 2016-10-29T15:02:00Z

etc/spack/defaults/config.yaml

+  # Locations where different types of modules should be installed.
+  module_roots:
+    tcl:    $spack/share/spack/modules
+    lmod:   $spack/share/spack/lmod


The Spack package for this is called environment-modules; I would suggest the same name be used here, rather than tcl.

I would not, as:

this is consistent with the format in modules.yaml

tcl identifies the type of module files written rather than the tools that understand them

The case for lmod was discussed in #107 and lmod was chosen over hierarchical_lua because of established conventions in the HPC communities.

As for (1)... if it were changed, it would have to be changed everywhere.

Personally I like the tcl designator, mainly because everything under modules: is technically an "environment module". But lmod, tcl, and dotkit is how most people (at least at LLNL) refer to them.

Probably should put parenthetically the first couple of times tcl is mentioned, that it's really environment-modules.

citibeth · 2016-10-29T15:02:51Z

etc/spack/defaults/config.yaml

+
+  # Cache directory already downloaded source tarballs and archived
+  # repositories. This can be purged with spack purge
+  source_cache: $spack/var/spack/cache


I think spack cache purge or spack purge cache would be more descriptive than spack purge

citibeth · 2016-10-29T15:03:09Z

etc/spack/defaults/config.yaml

+
+  # Cache directory for miscellaneous files, like the package index.
+  misc_cache: ~/.spack/cache
+


Is this also purged with spack purge?

Good point. I should put the right options to Spack purge on there. Yes, you can purge this with spack purge -u/--user-cache. That's kind of inconsistent with the naming here, so I'd say we should change the argument to spack purge -m/--misc-cache, since it could go someplace other than the user home directory. The distinction is really that the stuff in this cache is long-lived and small, so it doesn't need to be purged. The stuff in the build stage is short-lived, large, and purging doesn't hurt.

We need better terminology, the word "cache" is inherently confusing (since there are two caches). How about spack purge --misc-cache and spack purge download-cache?

citibeth · 2016-10-29T15:21:48Z

lib/spack/docs/configuration.rst

+   higher precedence scopes, as defaults may change from version to
+   version.
+
+2. **site**: Stored in ``$(prefix)/etc/spack/``.  Settings here affect


So... the file is not copied from defaults to site?

No. The files have never been copied...

Needs to be made clear in the docs...

citibeth · 2016-10-29T15:27:04Z

lib/spack/docs/configuration.rst

+
+3. **user**: Stored in the home directory, ``~/.spack/``. These settings
+   affect all instances of Spack and take the highest precedence.
+


We also need a project scope. Typical use case... I'm building two software stacks with different version/dependency selections in them. Or I want to run an integration test with a packages.yaml different from my "normal" one. Suggested ways to find the project scope would be:

An env var

A command line argument. (This would have to be available on all commands, ideally parsed and set up before the command is run. So then one could alias myproject-spack 'spack --config ~/myproject/dotspack'

Search up the source tree for a dotspack directory.

I would lean toward (2) as probably the best way to do this. As long as the Bash alias works, users can easily set up as many per-project Spack-like commands as they need.

I agree but this is a new feature not on the list for 0.10 and we should save it for another PR.

citibeth · 2016-10-29T15:27:50Z

lib/spack/docs/configuration.rst

+Commands that modify scopes (e.g., ``spack compilers``, ``spack repo``,
+etc.) take a ``--scope=<name>`` parameter that you can use to control
+which scope is modified.
+


Do any commands automagically modify scope? For example, the first time you run Spack? In that case, what scope will be modified? Should Spack ask for confirmation before modifying it?

The compiler detection is run the first time you start Spack, and compilers.yaml is put in the highest precedence scope available.

I don't think that should ask confirmation because first-time users have no idea what that means.

OK, no asking for confirmation. It still needs to be decided (and documented) which scope is modified when there is a command that automagically modifies configuration.

citibeth · 2016-10-29T15:29:50Z

lib/spack/docs/configuration.rst

+4. ``site/<platform>``
+5. ``user``
+6. ``user/<platform>``
+


How do we tell what <platform> is for our configuration?

Are any platform-specific files included in defaults?

With per-project configurations, user/<platform> might be less necessary. But it's nice to have options...

I can make this clearer.

citibeth · 2016-10-29T15:30:41Z

lib/spack/docs/configuration.rst

+scopes, Spack will merge the settings between the two files.  You can see
+the results of this merge with the ``spack config get <configtype>``
+command. For example, if your configurations look like this:
+


Most of this stuff has been there -- it just wasn't documented outside config.py. I realized that it probably needed to go in the real docs, since the config system in Spack is actually halfway decent at taking options from many different places/roles and combining them.

citibeth · 2016-10-29T15:31:51Z

I look forward to seeing this merged (if nothing else because it touches so many files). The biggest thing I think is messing is per-project scope. I really think that needs to go in before this is merged.

tgamblin · 2016-10-30T23:10:38Z

Ok, @citibeth brought up some good points about the docs. I've reorganized them and tried to address all the concerns -- I think things are looking much better.

New stuff:

Refactored configuration section of the reference manual:
- Top-level overview of how the config files and precedence work
- "Basic settings" section on config.yaml
- "Build settings" section covering the various uses of packages.yaml. This also makes the flexibility of packages.yaml more prominent in the docs (I think it was kind of buried before).
- Modules stuff is all in one section. High level section is just called "Modules". @alalazo

Various refactors:

Reverted clean to dirty, because it was actually introducing a lot of negative logic because we usually talk about dirty things, not clean things. We're going to ask users "was your build dirty?" and not "was your build not clean?". And there was confusion with spack.clean and spack clean. I think it's clearer to leave it dirty.
Refactored clean and dirty args into spack.cmd.common.arguments.
Refactored spack.cmd.common.arguments so that the arguments are written the same way they are for argparse, which should make it easier to move args into there.

I think that is more than enough for one PR. FWIW, spack repo and repos.yaml still need to be documented a bit and needs a reference guide section. That can wait for another PR.

I'd like to get this merged soon so try to get any additional requests in soon.

citibeth

LGTM, only minor docs suggestions.

citibeth · 2016-10-31T00:20:00Z

lib/spack/docs/configuration.rst

+etc.) take a ``--scope=<name>`` parameter that you can use to control
+which scope is modified.  By default they modify the highest-precedence
+scope.
+


...meaning, they modify the highest-precedence scope that exists? So if you don't have a user scope, it will modify the site scope? This should be made crystal clear. Also, I'm not sure it's the right choice; since clueless users can then accidentally modify the site scope. Probably best for config-modifying commands to just modify the user scope by default --- creating a new one if none exists.

citibeth · 2016-10-31T00:20:51Z

lib/spack/docs/configuration.rst

+
+When spack queries for configuration parameters, it searches in
+higher-precedence scopes first.  So, settings in ab higher-precedence
+file can override those with the same key in a lower-precedence one.  For


ab is typo?

citibeth · 2016-10-31T00:21:12Z

lib/spack/docs/configuration.rst

+file can override those with the same key in a lower-precedence one.  For
+list-valued settings, Spack *prepends* higher-precedence settings to
+lower-precedence settings.
+


How do you delete things from lower-precedence scope? If deleting is not yet possible, that should probably be noted in the docs. But I think it's a feature people will want at some point.

I see below that the double-colon notation is mentioned. It should probably be at least referenced right here, so readers aren't left on the hook wondering if deleting is possible.

citibeth · 2016-10-31T00:22:53Z

lib/spack/docs/configuration.rst

+override lower-precedence settings.  To do this, you can use *two* colons
+at the end of a key in a configuration file.  For example, if the
+**site** ``config.yaml`` above looks like this:
+


I see, this is how things can be "deleted". I think this makes sense to me, although the double-colon notation is somewhat subtle (but maybe standard YAML?)

- Added a schema for config.yaml - Moved install tree configuration to config.yaml - Moved etc/spack/install.yaml to etc/spack/defaults/config.yaml - renamed install_area to "store", to use a term in common with guix/nix. - in `config.yaml` file, it's called the `install_tree` to be more intuitive to users. - `install_tree` might've worked in the code, but `install_tree` is already a global function in the spack namespace, from llnl.util.filesystem.

- Moved temp finding logic to spack.stage - Updated stage tests - Added tests for new path substaitution of $user, $spack, $tempdir

- Added new preferences to config.yaml: - source_cache - misc_cache - verify_ssl - checksum - dirty

- generalized and fixed to work with any key in YAML file - simplified schema writing, as well - add more unit tests for the config system - Rename test/yaml.py to test/spack_yaml.py - Add test/yaml.pyc to ignored pyc files.

tgamblin · 2016-10-31T06:54:45Z

Ok this is (finally) done. This PR actually resulted in a lot more changes than @hegner's original, but it's a bunch of changes that Spack really needed. The config system is improved and had some bugs fixed, and things are documented much more thoroughly.

One thing that remains unfinished here is the default config scope behavior. Comment on #2184 if you care about that one -- I didn't feel like holding this up more for that.

tgamblin added configuration WIP labels Oct 27, 2016

tgamblin mentioned this pull request Oct 27, 2016

Make install directory a config option #908

Closed

tgamblin force-pushed the config_yaml branch 2 times, most recently from 3b4a74f to ad61950 Compare October 28, 2016 19:54

tgamblin mentioned this pull request Oct 28, 2016

trouble using spack on NERSC edison platform. #2166

Closed

adamjstewart reviewed Oct 29, 2016

View reviewed changes

tgamblin force-pushed the config_yaml branch from aaf456e to b8068d9 Compare October 29, 2016 02:53

citibeth reviewed Oct 29, 2016

View reviewed changes

tgamblin added the documentation Improvements or additions to documentation label Oct 30, 2016

tgamblin force-pushed the config_yaml branch 2 times, most recently from 8a1fbde to 1e07e4a Compare October 30, 2016 23:07

citibeth reviewed Oct 31, 2016

View reviewed changes

tgamblin force-pushed the config_yaml branch 2 times, most recently from 3d52c24 to ee4991d Compare October 31, 2016 01:36

tgamblin mentioned this pull request Oct 31, 2016

What config scope should be the default? #2184

Open

tgamblin force-pushed the config_yaml branch from ee4991d to c99c02c Compare October 31, 2016 03:51

tgamblin mentioned this pull request Oct 31, 2016

Removes cyclic dependency on spack.config. #2121

Merged

hegner and others added 10 commits October 30, 2016 21:03

refactor settings for install area and make them config options

d41d13e

add documentation for new option to set install area

7212bb2

Move temp directory configuration to config.yaml

84d286b

- Moved temp finding logic to spack.stage - Updated stage tests - Added tests for new path substaitution of $user, $spack, $tempdir

Add module_roots to config.yaml

a6bfe4a

source_cache, misc_cache, verify_ssl, checksum, & dirty in config.yaml

1857e7b

- Added new preferences to config.yaml: - source_cache - misc_cache - verify_ssl - checksum - dirty

Make provider_index use spack yaml for cleaner output.

72a176c

travis.yml: Only build master and develop on push

4310799

Add spack arch --platform option

ec98e17

Documentation for configuration

30c8457

tgamblin force-pushed the config_yaml branch from c99c02c to 1d148d0 Compare October 31, 2016 04:33

tgamblin added 2 commits October 30, 2016 23:33

Bugfix: '::' only worked on top-level key in config.

29b90e8

- generalized and fixed to work with any key in YAML file - simplified schema writing, as well - add more unit tests for the config system - Rename test/yaml.py to test/spack_yaml.py - Add test/yaml.pyc to ignored pyc files.

Allow common args to be written the same way regular args are.

1d9e080

tgamblin force-pushed the config_yaml branch from 1d148d0 to 1d9e080 Compare October 31, 2016 06:34

tgamblin merged commit 4be703c into develop Oct 31, 2016

This was referenced Oct 31, 2016

Spack --dirty in configuration #1925

Closed

Use JSON for the database instead of YAML. #2189

Merged

adamjstewart mentioned this pull request Nov 9, 2016

Allow prefix != spack source tree #126

Closed

tgamblin deleted the config_yaml branch December 31, 2016 04:59


		# Cache directory for miscellaneous files, like the package index.
		misc_cache: ~/.spack/cache


		3. user: Stored in the home directory, ``~/.spack/``. These settings
		affect all instances of Spack and take the highest precedence.

Add config.yaml file #2152

Add config.yaml file #2152

Conversation

tgamblin commented Oct 27, 2016 • edited

adamjstewart commented Oct 27, 2016

hegner commented Oct 27, 2016

tgamblin commented Oct 27, 2016

adamjstewart commented Oct 27, 2016

tgamblin commented Oct 27, 2016

adamjstewart commented Oct 27, 2016

citibeth commented Oct 27, 2016

tgamblin commented Oct 28, 2016

alalazo commented Oct 28, 2016

citibeth commented Oct 28, 2016

tgamblin commented Oct 28, 2016

adamjstewart commented Oct 28, 2016

citibeth commented Oct 28, 2016

pramodk commented Oct 28, 2016

tgamblin commented Oct 28, 2016 • edited

pramodk commented Oct 28, 2016

tgamblin commented Oct 28, 2016

tgamblin commented Oct 28, 2016 • edited

Choose a reason for hiding this comment

tgamblin commented Oct 29, 2016

tgamblin commented Oct 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

citibeth Oct 30, 2016 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

citibeth commented Oct 29, 2016

tgamblin commented Oct 30, 2016

citibeth left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tgamblin commented Oct 31, 2016

tgamblin commented Oct 27, 2016 •

edited

tgamblin commented Oct 28, 2016 •

edited

tgamblin commented Oct 28, 2016 •

edited

citibeth Oct 30, 2016 •

edited