`spack install --monitor` fails with `Error: 'full_hash'` #29704

iarspider · 2022-03-24T16:12:44Z

Steps to reproduce

$ spack env create test spack.yaml
$ spack env activate test
$ spack add pacparser
$ spack concretize -f
$ spack -d --stacktrace install --monitor --monitor-save-local

Error message

lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.150671] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.198803] Reading config file /build/razumov/vanilla_spack/var/spack/environments/test/spack.yaml
lib/spack/spack/environment/environment.py:161 ==> [2022-03-25-12:19:23.202212] Using environment 'test'
lib/spack/spack/cmd/__init__.py:123 ==> [2022-03-25-12:19:23.209235] Imported install from built-in commands
lib/spack/spack/cmd/__init__.py:123 ==> [2022-03-25-12:19:23.211429] Imported install from built-in commands
lib/spack/spack/util/executable.py:197 ==> [2022-03-25-12:19:23.215147] '/usr/bin/git' '-C' '/build/razumov/vanilla_spack' 'rev-parse' 'HEAD'
lib/spack/spack/util/executable.py:197 ==> [2022-03-25-12:19:23.224073] '/usr/bin/git' '-C' '/build/razumov/vanilla_spack' 'rev-parse' 'HEAD'
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.249451] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/environment/environment.py:185 ==> [2022-03-25-12:19:23.264793] Deactivated environment 'test'
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.288838] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/config.yaml
lib/spack/spack/database.py:377 ==> [2022-03-25-12:19:23.324522] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:381 ==> [2022-03-25-12:19:23.324871] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.326162] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/database.py:377 ==> [2022-03-25-12:19:23.340424] DATABASE LOCK TIMEOUT: 3s
lib/spack/spack/database.py:381 ==> [2022-03-25-12:19:23.340727] PACKAGE LOCK TIMEOUT: No timeout
lib/spack/spack/bootstrap.py:585 ==> [2022-03-25-12:19:23.341083] [BOOTSTRAP CONFIG SCOPE] name=_builtin
lib/spack/spack/bootstrap.py:601 ==> [2022-03-25-12:19:23.341946] [BOOTSTRAP CONFIG SCOPE] name=defaults, path=/build/razumov/vanilla_spack/etc/spack/defaults
lib/spack/spack/bootstrap.py:602 ==> [2022-03-25-12:19:23.342165] [BOOTSTRAP CONFIG SCOPE] name=defaults/linux, path=/build/razumov/vanilla_spack/etc/spack/defaults/linux
lib/spack/spack/bootstrap.py:601 ==> [2022-03-25-12:19:23.342414] [BOOTSTRAP CONFIG SCOPE] name=bootstrap, path=/afs/cern.ch/user/r/razumov/.spack/bootstrap/config
lib/spack/spack/bootstrap.py:602 ==> [2022-03-25-12:19:23.342603] [BOOTSTRAP CONFIG SCOPE] name=bootstrap/linux, path=/afs/cern.ch/user/r/razumov/.spack/bootstrap/config/linux
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.364607] Reading config file /afs/cern.ch/user/r/razumov/.spack/bootstrap/config/linux/compilers.yaml
lib/spack/spack/bootstrap.py:741 ==> [2022-03-25-12:19:23.428454] [BOOTSTRAP ROOT SPEC] clingo-bootstrap@spack+python %gcc target=x86_64
lib/spack/spack/bootstrap.py:484 ==> [2022-03-25-12:19:23.428743] [BOOTSTRAP MODULE clingo] Try importing from Python
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.429592] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/bootstrap.yaml
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.685369] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/config.yaml
lib/spack/spack/bootstrap.py:89 ==> [2022-03-25-12:19:23.823281] [BOOTSTRAP MODULE clingo] The installed spec "clingo-bootstrap@spack+python %gcc target=x86_64 ^python@3.6 /vcipwnf57slgoo7busvvkzjkk7vydeb5" provides the "clingo" Python module
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.825095] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/config.yaml
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.865316] Reading config file /build/razumov/vanilla_spack/var/spack/environments/test/spack.yaml
lib/spack/spack/environment/environment.py:161 ==> [2022-03-25-12:19:23.868830] Using environment 'test'
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.869253] Reading config file /build/razumov/vanilla_spack/etc/spack/defaults/repos.yaml
lib/spack/spack/config.py:1016 ==> [2022-03-25-12:19:23.962567] Reading config file /build/razumov/vanilla_spack/etc/spack/compilers.yaml
spec= pacparser@1.3.7%gcc@9.3.0~python arch=linux-centos7-broadwell
Traceback (most recent call last):
  File "/build/razumov/vanilla_spack/bin/spack", line 98, in <module>
    sys.exit(spack.main.main())
  File "/build/razumov/vanilla_spack/lib/spack/spack/main.py", line 916, in main
    return _main(argv)
  File "/build/razumov/vanilla_spack/lib/spack/spack/main.py", line 871, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/build/razumov/vanilla_spack/lib/spack/spack/main.py", line 899, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/build/razumov/vanilla_spack/lib/spack/spack/main.py", line 551, in _invoke_command
    return_val = command(parser, args)
  File "/build/razumov/vanilla_spack/lib/spack/spack/cmd/install.py", line 359, in install
    monitor.new_configuration(specs)
  File "/build/razumov/vanilla_spack/lib/spack/spack/monitor.py", line 416, in new_configuration
    as_dict = {"spec": spec.to_dict(hash=ht.full_hash)['spec'],
  File "/build/razumov/vanilla_spack/lib/spack/spack/spec.py", line 2020, in to_dict
    spec_hash = s.node_dict_with_hashes(hash)[hash.name]
KeyError: 'full_hash'

Information on your system

Spack: 0.18.0.dev0
Python: 3.6.8
Platform: linux-centos7-broadwell
Concretizer: clingo

General information

I have run spack debug report and reported the version of Spack/Python/Platform
I have searched the issues of this repo and believe this is not a duplicate
I have run the failing commands in debug mode and reported the output

The text was updated successfully, but these errors were encountered:

iarspider · 2022-03-24T16:13:37Z

@vsoch ping

vsoch · 2022-03-24T16:23:31Z

The full hash needs to be in a spec for spack monitor to receive it. If someone removed or changed it, it needs to be added back.

iarspider · 2022-03-24T17:12:12Z

I need a better reproducer - this one was created by dumping the spec with missing full_hash in a much bigger environment (~460 specs)

iarspider · 2022-03-24T21:20:00Z

Here is full spec: https://cernbox.cern.ch/index.php/s/RGtCwNtFBw0aMQR . And the lock file after doing spack concretize: https://cernbox.cern.ch/index.php/s/llQLQQg2iZ6QHGb .

vsoch · 2022-03-24T21:21:13Z

Why is the spec in yaml? Isn’t that a really old format?

iarspider · 2022-03-24T21:21:19Z

To debug, I have made two changes:

index c0df4ea..d84c598 100644
--- a/lib/spack/spack/monitor.py
+++ b/lib/spack/spack/monitor.py
@@ -405,8 +405,12 @@ def new_configuration(self, specs):
                 spec.concretize()

             # Remove extra level of nesting
-            as_dict = {"spec": spec.to_dict(hash=ht.full_hash)['spec'],
+            try:
+                as_dict = {"spec": spec.to_dict(hash=ht.full_hash)['spec'],
                        "spack_version": self.spack_version}
+            except KeyError:
+                print(spec)
+                raise

             if self.save_local:
                 filename = "spec-%s-%s-config.json" % (spec.name, spec.version)
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
index 8e1e81c..83337e5 100644
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -319,6 +319,9 @@ def _flush_callbacks(check_name):
                             if phase is not None:
                                 break

+                        if phase is None:
+                            print(checks)
+                            raise RuntimeError('foo')
                         phase = attr_dict[phase_attr] = phase.copy()
                     getattr(phase, check_name).extend(funcs)
                 # Clear the attribute for the next class

and got the following output:

{'arch': {'platform': 'linux',
          'platform_os': 'centos7',
          'target': {'features': ['adx',
                                  'aes',
                                  'avx',
                                  'avx2',
                                  'bmi1',
                                  'bmi2',
                                  'f16c',
                                  'fma',
                                  'mmx',
                                  'movbe',
                                  'pclmulqdq',
                                  'popcnt',
                                  'rdrand',
                                  'rdseed',
                                  'sse',
                                  'sse2',
                                  'sse4_1',
                                  'sse4_2',
                                  'ssse3'],
                     'generation': 0,
                     'name': 'broadwell',
                     'parents': ['haswell'],
                     'vendor': 'GenuineIntel'}},
 'build_hash': 'z4hq7ipcuez6fjfuu5wxar5kvvwpplmi',
 'compiler': {'name': 'gcc', 'version': '9.3.0'},
 'dependencies': [{'full_hash': '6cbroemueau5rth4ij3qoe4q7l2kisqc',
                   'name': 'cmake',
                   'type': ['build']}],
 'hash': 'gyqo4he6ee5p3elavffqsgwkvudnt5rp',
 'name': 'abseil-cpp',
 'namespace': 'builtin',
 'package_hash': '4uwi4php6wpnpglylh5hvb6pw6nqx7w4556bp5b6t4dfyu6u3j3a====',
 'parameters': {'build_type': 'Release',
                'cflags': [],
                'cppflags': [],
                'cxxflags': [],
                'cxxstd': '17',
                'fflags': [],
                'ipo': False,
                'ldflags': [],
                'ldlibs': [],
                'shared': True},
 'version': '20210324.2'}
abseil-cpp@20210324.2%gcc@9.3.0~ipo+shared build_type=Release cxxstd=17 arch=linux-centos7-broadwell ^autoconf@2.69%gcc@9.3.0 arch=linux-centos7-broadwell ^automake@1.16.1%gcc@9.3.0 arch=linux-centos7-broadwell ^bzip2@1.0.6%gcc@9.3.0~debug~pic+shared arch=linux-centos7-broadwell ^cmake@3.21.4%gcc@9.3.0~doc~ncurses+openssl~ownlibs~qt build_type=Release arch=linux-centos7-broadwell ^curl@7.79.0%gcc@9.3.0+gssapi~ldap~libidn2~librtmp~libssh~libssh2~nghttp2 tls=openssl arch=linux-centos7-broadwell ^diffutils@3.3%gcc@9.3.0 arch=linux-centos7-broadwell ^expat@2.1.0%gcc@9.3.0~libbsd arch=linux-centos7-broadwell ^krb5@1.15.1%gcc@9.3.0+shared patches=3d75052730690579484e65a5bf0f92f6c3b20d9c43a413862d087774f431d9e9 arch=linux-centos7-broadwell ^libarchive@3.5.2%gcc@9.3.0~iconv compression=bz2lib,lz4,lzma,lzo2,zlib crypto=openssl libs=shared,static programs=none xar=libxml2 arch=linux-centos7-broadwell ^libc@1.0%gcc@9.3.0+iconv~rpc arch=linux-centos7-broadwell ^libtool@2.4.6%gcc@9.3.0 arch=linux-centos7-broadwell ^libuv@1.42.0%gcc@9.3.0 arch=linux-centos7-broadwell ^libxml2@2.9.10%gcc@9.3.0~python arch=linux-centos7-broadwell ^lz4@1.9.3%gcc@9.3.0 libs=shared,static arch=linux-centos7-broadwell ^lzo@2.10%gcc@9.3.0 libs=shared,static arch=linux-centos7-broadwell ^m4@1.4.18%gcc@9.3.0~sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=linux-centos7-broadwell ^openssl@1.0.2k-fips%gcc@9.3.0~docs certs=system arch=linux-centos7-broadwell ^perl@5.16.3%gcc@9.3.0~cpanm+shared+threads patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac,3bbd7d6f9933d80b9571533867b444c6f8f5a1ba0575bfba1fba4db9d885a71a arch=linux-centos7-broadwell ^pkg-config@0.29.2%gcc@9.3.0+internal_glib arch=linux-centos7-broadwell ^rhash@1.4.2%gcc@9.3.0 patches=093518cee1195b20bd7b4f46b57c6e91906134985eeb3de87cd96a42e8a1db3e,3fbfe4603d2ec5228fd198fc87ff3ee281e1f68d252c1afceaa15cba76e9b6b4 arch=linux-centos7-broadwell ^xz@5.2.4%gcc@9.3.0~pic libs=shared arch=linux-centos7-broadwell ^zlib@1.2.11.cms%gcc@9.3.0+optimize+pic+shared arch=linux-centos7-broadwell

vsoch · 2022-03-24T21:22:22Z

Yeah I guess someone renamed full_hash to hash without thinking about spack monitor.

iarspider · 2022-03-24T21:23:16Z

Why is the spec in yaml? Isn’t that a really old format?

IIRC, the lock file was changed to JSON, the environment itself is still YAML

vsoch · 2022-03-24T21:23:22Z

You could try just renaming the index to use that instead.

iarspider · 2022-03-24T21:26:38Z

Yeah I guess someone renamed full_hash to hash without thinking about spack monitor.

I thought hash was dag_hash, which is different from full_hash. (grumble) that stoopid slack where history gets cut off at the worst time (/grumble)

vsoch · 2022-03-24T21:32:07Z

well, for spack monitor's sake it really just needs a unique id for the spec. It was a pretty arbitrary decision when it was first made. Technically we can present any of the hashes (changed and newly deemed worthy) for it to use. If the key is wrong for the server, the easiest thing to do will be just to add full_hash to be whatever spack is now using (e.g., dag).

iarspider · 2022-03-25T11:18:10Z

OK, I have a better reproducer. Let me update the first message...

iarspider · 2022-03-25T11:37:01Z

So, IIUC, full_hash is not written into spack.lock file by default (why?). But a spec read from spack.lock is considered final and it's full_hash is not recalculated (or, at least, not recalculated at the moment when spack monitor code is being called).

vsoch · 2022-03-25T16:43:54Z

That should be okay, try changing:

- as_dict = {"spec": spec.to_dict(hash=ht.full_hash)['spec'],
+ as_dict = {"spec": spec.to_dict(hash=ht.hash)['spec'],

or choose any hash that you want that you have. I'm not sure ht.hash is an option, maybe ht.dag_hash is, for example.

iarspider · 2022-05-19T16:53:23Z

Since spack monitor is going away, I'm closing this issue.

iarspider added bug Something isn't working triage The issue needs to be prioritized labels Mar 24, 2022

iarspider closed this as completed May 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`spack install --monitor` fails with `Error: 'full_hash'` #29704

`spack install --monitor` fails with `Error: 'full_hash'` #29704

iarspider commented Mar 24, 2022 •

edited

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 25, 2022

iarspider commented Mar 25, 2022

vsoch commented Mar 25, 2022

iarspider commented May 19, 2022

spack install --monitor fails with Error: 'full_hash' #29704

spack install --monitor fails with Error: 'full_hash' #29704

Comments

iarspider commented Mar 24, 2022 • edited

Steps to reproduce

Error message

Information on your system

General information

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 24, 2022

vsoch commented Mar 24, 2022

iarspider commented Mar 25, 2022

iarspider commented Mar 25, 2022

vsoch commented Mar 25, 2022

iarspider commented May 19, 2022

`spack install --monitor` fails with `Error: 'full_hash'` #29704

`spack install --monitor` fails with `Error: 'full_hash'` #29704

iarspider commented Mar 24, 2022 •

edited