Spurious local failures of test_install_commit and test_versions_from_git

[alalazo]

Steps to reproduce

Running unit tests locally:

$ spack unit-tests

might result in two tests failing at times on feature branches. Re-running the same tests like:

$ spack unit-test --lf

may result in having them PASSED. So far they does seem to always pass in CI.

Error message

$ spack unit-test

[ ... ]
================================================================= slowest 30 test durations ==================================================================
187.83s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_with_env
142.00s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_bootstrap_gcc
121.75s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_bootstrap_artifacts_buildcache
121.63s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_with_workarounds
101.88s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_read_broken_specs_url
101.55s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_override_runner_attrs
61.38s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_with_external_pkg
61.17s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_for_pr_pipeline
61.13s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_pkg_with_deps
51.08s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_with_cdash_token
41.08s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_temp_storage_url
40.89s call     lib/spack/spack/test/cmd/ci.py::test_ci_generate_with_custom_scripts
15.16s call     lib/spack/spack/test/cmd/versions.py::test_no_versions
13.47s call     lib/spack/spack/test/package_sanity.py::test_get_all_packages
8.86s call     lib/spack/spack/test/spec_semantics.py::test_is_extension_after_round_trip_to_dict[hdf5]
7.42s call     lib/spack/spack/test/cmd/gpg.py::test_gpg
7.01s call     lib/spack/spack/test/cmd/versions.py::test_versions_no_url
6.56s call     lib/spack/spack/test/install.py::test_install_times[mock_archive0]
6.49s call     lib/spack/spack/test/test_suite.py::test_test_ensure_stage
6.19s call     lib/spack/spack/test/spec_semantics.py::test_is_extension_after_round_trip_to_dict[git]
5.20s call     lib/spack/spack/test/cmd/env.py::test_stack_view_multiple_views[mock_archive0]
4.90s call     lib/spack/spack/test/cmd/env.py::test_stack_view_no_activate_without_default[mock_archive0]
4.86s call     lib/spack/spack/test/cmd/env.py::test_stack_view_activate_from_default[mock_archive0]
4.57s call     lib/spack/spack/test/cmd/env.py::test_view_link_roots[mock_archive0]
4.37s call     lib/spack/spack/test/cmd/env.py::test_stack_combinatorial_view[mock_archive0]
4.35s call     lib/spack/spack/test/cmd/env.py::test_stack_view_exclude[mock_archive0]
4.32s call     lib/spack/spack/test/cmd/env.py::test_stack_view_select[mock_archive0]
4.31s call     lib/spack/spack/test/cmd/env.py::test_view_link_all[mock_archive0]
4.24s call     lib/spack/spack/test/cmd/env.py::test_stack_view_select_and_exclude[mock_archive0]
4.08s call     lib/spack/spack/test/package_sanity.py::test_package_version_consistency
========================================================================== FAILURES ==========================================================================
___________________________________________________________________ test_versions_from_git ___________________________________________________________________

mock_git_version_info = ('/tmp/pytest-of-culpo/pytest-10/test_versions_from_git0/git_repo', 'file.txt', ['c20c6376f728ab7585ca8dc1ee25ff158bd3...7fea5a9e6c3209e6a910d8a', '83c9a484a124d1a1cb82364cf1c19f9a1fc6a14a', '2840933debab120e10288ace43ddc6f02f2e5cfe', ...])
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f3fce4802e0>, mock_packages = <spack.repo.RepoPath object at 0x7f3fce1594f0>

    def test_versions_from_git(mock_git_version_info, monkeypatch, mock_packages):
        repo_path, filename, commits = mock_git_version_info
        monkeypatch.setattr(spack.package.PackageBase, 'git', 'file://%s' % repo_path,
                            raising=False)
    
        for commit in commits:
>           spec = spack.spec.Spec('git-test-commit@%s' % commit)

lib/spack/spack/test/versions.py:594: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
lib/spack/spack/spec.py:1119: in __init__
    spec_list = SpecParser(self).parse(spec_like)
lib/spack/spack/parse.py:153: in parse
    return self.do_parse()
lib/spack/spack/spec.py:4717: in do_parse
    spec.version.generate_commit_lookup(pkg)
lib/spack/spack/version.py:489: in generate_commit_lookup
    self.commit_lookup.get(self.string)
lib/spack/spack/version.py:1045: in get
    self.data[commit] = self.lookup_commit(commit)
lib/spack/spack/version.py:1073: in lookup_commit
    self.fetcher.git("fetch", '--tags')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <exe: ['/usr/bin/git', '-c', 'advice.detachedHead=false']>, args = ('fetch', '--tags'), kwargs = {}, env_arg = None
env = {'CI_JOB_URL': 'https://some.domain/group/project/-/jobs/42', 'CI_PIPELINE_ID': '7192', 'CI_PIPELINE_URL': 'https://some.domain/group/project/-/pipelines/7', 'COLORTERM': 'truecolor', ...}
EnvironmentModifications = <class 'spack.util.environment.EnvironmentModifications'>, extra_env = {}, fail_on_error = True, ignore_errors = ()
ignore_quotes = False, input = None, output = None, error = None

    def __call__(self, *args, **kwargs):
        """Run this executable in a subprocess.
    
            Parameters:
                *args (str): Command-line arguments to the executable to run
    
            Keyword Arguments:
                _dump_env (dict): Dict to be set to the environment actually
                    used (envisaged for testing purposes only)
                env (dict or EnvironmentModifications): The environment with which
                    to run the executable
                extra_env (dict or EnvironmentModifications): Extra items to add to
                    the environment (neither requires nor precludes env)
                fail_on_error (bool): Raise an exception if the subprocess returns
                    an error. Default is True. The return code is available as
                    ``exe.returncode``
                ignore_errors (int or list): A list of error codes to ignore.
                    If these codes are returned, this process will not raise
                    an exception even if ``fail_on_error`` is set to ``True``
                ignore_quotes (bool): If False, warn users that quotes are not needed
                    as Spack does not use a shell. Defaults to False.
                input: Where to read stdin from
                output: Where to send stdout
                error: Where to send stderr
    
            Accepted values for input, output, and error:
    
            * python streams, e.g. open Python file objects, or ``os.devnull``
            * filenames, which will be automatically opened for writing
            * ``str``, as in the Python string type. If you set these to ``str``,
              output and error will be written to pipes and returned as a string.
              If both ``output`` and ``error`` are set to ``str``, then one string
              is returned containing output concatenated with error. Not valid
              for ``input``
            * ``str.split``, as in the ``split`` method of the Python string type.
              Behaves the same as ``str``, except that value is also written to
              ``stdout`` or ``stderr``.
    
            By default, the subprocess inherits the parent's file descriptors.
    
            """
        # Environment
        env_arg = kwargs.get('env', None)
    
        # Setup default environment
        env = os.environ.copy() if env_arg is None else {}
        self.default_envmod.apply_modifications(env)
        env.update(self.default_env)
    
        from spack.util.environment import EnvironmentModifications  # no cycle
    
        # Apply env argument
        if isinstance(env_arg, EnvironmentModifications):
            env_arg.apply_modifications(env)
        elif env_arg:
            env.update(env_arg)
    
        # Apply extra env
        extra_env = kwargs.get('extra_env', {})
        if isinstance(extra_env, EnvironmentModifications):
            extra_env.apply_modifications(env)
        else:
            env.update(extra_env)
    
        if '_dump_env' in kwargs:
            kwargs['_dump_env'].clear()
            kwargs['_dump_env'].update(env)
    
        fail_on_error = kwargs.pop('fail_on_error', True)
        ignore_errors = kwargs.pop('ignore_errors', ())
        ignore_quotes = kwargs.pop('ignore_quotes', False)
    
        # If they just want to ignore one error code, make it a tuple.
        if isinstance(ignore_errors, int):
            ignore_errors = (ignore_errors, )
    
        input  = kwargs.pop('input',  None)
        output = kwargs.pop('output', None)
        error  = kwargs.pop('error',  None)
    
        if input is str:
            raise ValueError('Cannot use `str` as input stream.')
    
        def streamify(arg, mode):
            if isinstance(arg, string_types):
                return open(arg, mode), True
            elif arg in (str, str.split):
                return subprocess.PIPE, False
            else:
                return arg, False
    
        ostream, close_ostream = streamify(output, 'w')
        estream, close_estream = streamify(error,  'w')
        istream, close_istream = streamify(input,  'r')
    
        if not ignore_quotes:
            quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
            if quoted_args:
                tty.warn(
                    "Quotes in command arguments can confuse scripts like"
                    " configure.",
                    "The following arguments may cause problems when executed:",
                    str("\n".join(["    " + arg for arg in quoted_args])),
                    "Quotes aren't needed because spack doesn't use a shell. "
                    "Consider removing them.",
                    "If multiple levels of quotation are required, use "
                    "`ignore_quotes=True`.")
    
        cmd = self.exe + list(args)
    
        escaped_cmd = ["'%s'" % arg.replace("'", "'\"'\"'") for arg in cmd]
        cmd_line_string = " ".join(escaped_cmd)
        tty.debug(cmd_line_string)
    
        try:
            proc = subprocess.Popen(
                cmd,
                stdin=istream,
                stderr=estream,
                stdout=ostream,
                env=env)
            out, err = proc.communicate()
    
            result = None
            if output in (str, str.split) or error in (str, str.split):
                result = ''
                if output in (str, str.split):
                    outstr = text_type(out.decode('utf-8'))
                    result += outstr
                    if output is str.split:
                        sys.stdout.write(outstr)
                if error in (str, str.split):
                    errstr = text_type(err.decode('utf-8'))
                    result += errstr
                    if error is str.split:
                        sys.stderr.write(errstr)
    
            rc = self.returncode = proc.returncode
            if fail_on_error and rc != 0 and (rc not in ignore_errors):
                long_msg = cmd_line_string
                if result:
                    # If the output is not captured in the result, it will have
                    # been stored either in the specified files (e.g. if
                    # 'output' specifies a file) or written to the parent's
                    # stdout/stderr (e.g. if 'output' is not specified)
                    long_msg += '\n' + result
    
>               raise ProcessError('Command exited with status %d:' %
                                   proc.returncode, long_msg)
E                                  spack.util.executable.ProcessError: Command exited with status 1:
E                                      '/usr/bin/git' '-c' 'advice.detachedHead=false' 'fetch' '--tags'

lib/spack/spack/util/executable.py:225: ProcessError
------------------------------------------------------------------- Captured stdout setup --------------------------------------------------------------------
Initialized empty Git repository in /tmp/pytest-of-culpo/pytest-10/test_versions_from_git0/git_repo/.git/
[master (root-commit) 31ae5e8] first commit
 Date: Tue Sep 1 12:00:00 2020 +0200
 1 file changed, 1 insertion(+)
 create mode 100644 file.txt
[master 2ab3982] second commit
 Date: Wed Jan 1 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 7295a8d] first 1.x commit
 Date: Thu Jan 2 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 3eb2a11] second 1.x commit
 Date: Fri Jan 3 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 2840933] third main commit
 Date: Sat Jan 4 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 83c9a48] fourth main commit
 Date: Sun Jan 5 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 16105cb] third 1.x commit
 Date: Mon Jan 6 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x c20c637] fourth 1.x commit
 Date: Tue Jan 7 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
------------------------------------------------------------------- Captured stderr setup --------------------------------------------------------------------
Switched to a new branch '1.x'
Switched to branch 'master'
Switched to branch '1.x'
-------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------
warning: no common commits
From file:///tmp/pytest-of-culpo/pytest-10/test_versions_from_git0/git_repo
 * branch            HEAD       -> FETCH_HEAD
 ! [rejected]        1.2        -> 1.2  (would clobber existing tag)
 ! [rejected]        v1.0       -> v1.0  (would clobber existing tag)
 ! [rejected]        v1.1       -> v1.1  (would clobber existing tag)
 ! [rejected]        v2.0       -> v2.0  (would clobber existing tag)
____________________________________________________________________ test_install_commit _____________________________________________________________________

mock_git_version_info = ('/tmp/pytest-of-culpo/pytest-10/test_install_commit0/git_repo', 'file.txt', ['0fe2e1bd860f8f021fce38df8fa6037fd3a2242...2b6d2c83817673d24b8b7f6', '0324004f60f7eed73541a58ea7912cbcaecf7e57', '1061f03f8116dd7a8a1784f4b9c3b74a6f7326b2', ...])
install_mockery = None, mock_packages = <spack.repo.RepoPath object at 0x7f400f803ee0>
monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f402c7209d0>

    def test_install_commit(
            mock_git_version_info, install_mockery, mock_packages, monkeypatch):
        """
        Test installing a git package from a commit.
    
        This ensures Spack appropriately associates commit versions with their
        packages in time to do version lookups. Details of version lookup tested elsewhere
        """
        repo_path, filename, commits = mock_git_version_info
        monkeypatch.setattr(spack.package.PackageBase,
                            'git', 'file://%s' % repo_path,
                            raising=False)
    
        commit = commits[-1]
>       spec = spack.spec.Spec('git-test-commit@%s' % commit)

lib/spack/spack/test/cmd/install.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
lib/spack/spack/spec.py:1119: in __init__
    spec_list = SpecParser(self).parse(spec_like)
lib/spack/spack/parse.py:153: in parse
    return self.do_parse()
lib/spack/spack/spec.py:4717: in do_parse
    spec.version.generate_commit_lookup(pkg)
lib/spack/spack/version.py:489: in generate_commit_lookup
    self.commit_lookup.get(self.string)
lib/spack/spack/version.py:1045: in get
    self.data[commit] = self.lookup_commit(commit)
lib/spack/spack/version.py:1073: in lookup_commit
    self.fetcher.git("fetch", '--tags')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <exe: ['/usr/bin/git', '-c', 'advice.detachedHead=false']>, args = ('fetch', '--tags'), kwargs = {}, env_arg = None
env = {'CC': '/home/culpo/PycharmProjects/spack/lib/spack/env/gcc/gcc', 'CI_JOB_URL': 'https://some.domain/group/project/-/jobs/42', 'CI_PIPELINE_ID': '7192', 'CI_PIPELINE_URL': 'https://some.domain/group/project/-/pipelines/7', ...}
EnvironmentModifications = <class 'spack.util.environment.EnvironmentModifications'>, extra_env = {}, fail_on_error = True, ignore_errors = ()
ignore_quotes = False, input = None, output = None, error = None

    def __call__(self, *args, **kwargs):
        """Run this executable in a subprocess.
    
            Parameters:
                *args (str): Command-line arguments to the executable to run
    
            Keyword Arguments:
                _dump_env (dict): Dict to be set to the environment actually
                    used (envisaged for testing purposes only)
                env (dict or EnvironmentModifications): The environment with which
                    to run the executable
                extra_env (dict or EnvironmentModifications): Extra items to add to
                    the environment (neither requires nor precludes env)
                fail_on_error (bool): Raise an exception if the subprocess returns
                    an error. Default is True. The return code is available as
                    ``exe.returncode``
                ignore_errors (int or list): A list of error codes to ignore.
                    If these codes are returned, this process will not raise
                    an exception even if ``fail_on_error`` is set to ``True``
                ignore_quotes (bool): If False, warn users that quotes are not needed
                    as Spack does not use a shell. Defaults to False.
                input: Where to read stdin from
                output: Where to send stdout
                error: Where to send stderr
    
            Accepted values for input, output, and error:
    
            * python streams, e.g. open Python file objects, or ``os.devnull``
            * filenames, which will be automatically opened for writing
            * ``str``, as in the Python string type. If you set these to ``str``,
              output and error will be written to pipes and returned as a string.
              If both ``output`` and ``error`` are set to ``str``, then one string
              is returned containing output concatenated with error. Not valid
              for ``input``
            * ``str.split``, as in the ``split`` method of the Python string type.
              Behaves the same as ``str``, except that value is also written to
              ``stdout`` or ``stderr``.
    
            By default, the subprocess inherits the parent's file descriptors.
    
            """
        # Environment
        env_arg = kwargs.get('env', None)
    
        # Setup default environment
        env = os.environ.copy() if env_arg is None else {}
        self.default_envmod.apply_modifications(env)
        env.update(self.default_env)
    
        from spack.util.environment import EnvironmentModifications  # no cycle
    
        # Apply env argument
        if isinstance(env_arg, EnvironmentModifications):
            env_arg.apply_modifications(env)
        elif env_arg:
            env.update(env_arg)
    
        # Apply extra env
        extra_env = kwargs.get('extra_env', {})
        if isinstance(extra_env, EnvironmentModifications):
            extra_env.apply_modifications(env)
        else:
            env.update(extra_env)
    
        if '_dump_env' in kwargs:
            kwargs['_dump_env'].clear()
            kwargs['_dump_env'].update(env)
    
        fail_on_error = kwargs.pop('fail_on_error', True)
        ignore_errors = kwargs.pop('ignore_errors', ())
        ignore_quotes = kwargs.pop('ignore_quotes', False)
    
        # If they just want to ignore one error code, make it a tuple.
        if isinstance(ignore_errors, int):
            ignore_errors = (ignore_errors, )
    
        input  = kwargs.pop('input',  None)
        output = kwargs.pop('output', None)
        error  = kwargs.pop('error',  None)
    
        if input is str:
            raise ValueError('Cannot use `str` as input stream.')
    
        def streamify(arg, mode):
            if isinstance(arg, string_types):
                return open(arg, mode), True
            elif arg in (str, str.split):
                return subprocess.PIPE, False
            else:
                return arg, False
    
        ostream, close_ostream = streamify(output, 'w')
        estream, close_estream = streamify(error,  'w')
        istream, close_istream = streamify(input,  'r')
    
        if not ignore_quotes:
            quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
            if quoted_args:
                tty.warn(
                    "Quotes in command arguments can confuse scripts like"
                    " configure.",
                    "The following arguments may cause problems when executed:",
                    str("\n".join(["    " + arg for arg in quoted_args])),
                    "Quotes aren't needed because spack doesn't use a shell. "
                    "Consider removing them.",
                    "If multiple levels of quotation are required, use "
                    "`ignore_quotes=True`.")
    
        cmd = self.exe + list(args)
    
        escaped_cmd = ["'%s'" % arg.replace("'", "'\"'\"'") for arg in cmd]
        cmd_line_string = " ".join(escaped_cmd)
        tty.debug(cmd_line_string)
    
        try:
            proc = subprocess.Popen(
                cmd,
                stdin=istream,
                stderr=estream,
                stdout=ostream,
                env=env)
            out, err = proc.communicate()
    
            result = None
            if output in (str, str.split) or error in (str, str.split):
                result = ''
                if output in (str, str.split):
                    outstr = text_type(out.decode('utf-8'))
                    result += outstr
                    if output is str.split:
                        sys.stdout.write(outstr)
                if error in (str, str.split):
                    errstr = text_type(err.decode('utf-8'))
                    result += errstr
                    if error is str.split:
                        sys.stderr.write(errstr)
    
            rc = self.returncode = proc.returncode
            if fail_on_error and rc != 0 and (rc not in ignore_errors):
                long_msg = cmd_line_string
                if result:
                    # If the output is not captured in the result, it will have
                    # been stored either in the specified files (e.g. if
                    # 'output' specifies a file) or written to the parent's
                    # stdout/stderr (e.g. if 'output' is not specified)
                    long_msg += '\n' + result
    
>               raise ProcessError('Command exited with status %d:' %
                                   proc.returncode, long_msg)
E                                  spack.util.executable.ProcessError: Command exited with status 1:
E                                      '/usr/bin/git' '-c' 'advice.detachedHead=false' 'fetch' '--tags'

lib/spack/spack/util/executable.py:225: ProcessError
------------------------------------------------------------------- Captured stdout setup --------------------------------------------------------------------
Initialized empty Git repository in /tmp/pytest-of-culpo/pytest-10/test_install_commit0/git_repo/.git/
[master (root-commit) e429553] first commit
 Date: Wed Jan 8 12:00:00 2020 +0300
 1 file changed, 1 insertion(+)
 create mode 100644 file.txt
[master be5ab0a] second commit
 Date: Thu Jan 9 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 2668d3f] first 1.x commit
 Date: Fri Jan 10 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 6dd30a6] second 1.x commit
 Date: Sat Jan 11 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 1061f03] third main commit
 Date: Sun Jan 12 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 0324004] fourth main commit
 Date: Mon Jan 13 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 9849704] third 1.x commit
 Date: Tue Jan 14 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 0fe2e1b] fourth 1.x commit
 Date: Wed Jan 15 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
------------------------------------------------------------------- Captured stderr setup --------------------------------------------------------------------
Switched to a new branch '1.x'
Switched to branch 'master'
Switched to branch '1.x'
-------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------
warning: no common commits
From file:///tmp/pytest-of-culpo/pytest-10/test_install_commit0/git_repo
 * branch            HEAD       -> FETCH_HEAD
 ! [rejected]        1.2        -> 1.2  (would clobber existing tag)
 ! [rejected]        v1.0       -> v1.0  (would clobber existing tag)
 ! [rejected]        v1.1       -> v1.1  (would clobber existing tag)
 ! [rejected]        v2.0       -> v2.0  (would clobber existing tag)
======================================== 2 failed, 2969 passed, 141 skipped, 6 xfailed, 3 xpassed in 1836.77 seconds =========================================

Information on your system

• Spack: 0.16.2-4312-8fceaf4a33
• Python: 3.8.10
• Platform: linux-ubuntu20.04-icelake
• Concretizer: clingo

General information

• I have run spack debug report and reported the version of Spack/Python/Platform
• I have searched the issues of this repo and believe this is not a duplicate
• I have run the failing commands in debug mode and reported the output

[alalazo]

@becker33 @vsoch I'm getting the two errors above consistently on my laptop on every branch I work on (not in CI though). I didn't investigate them yet, but I think they're related to #24639 so I wonder if anything rings a bell here.

[haampie]

spack unit-test lib/spack/spack/test/cmd/install.py

================================================================================ FAILURES =================================================================================
___________________________________________________________________________ test_install_commit ___________________________________________________________________________

mock_git_version_info = ('/tmp/pytest-of-harmen/pytest-23/test_install_commit0/git_repo', 'file.txt', ['27b83d106f96be7f2d1d50f625a4f478bcecef...85fceb7af4ebafc45511219', 'aff22cd288fbc08d967a6d15cf7fd7b88a7f9078', '58eb58ed69822e688e100d4be1101fa87705dd74', ...])
install_mockery = None, mock_packages = <spack.repo.RepoPath object at 0x7f87eea9e370>, monkeypatch = <_pytest.monkeypatch.MonkeyPatch object at 0x7f87ec136fd0>

    def test_install_commit(
            mock_git_version_info, install_mockery, mock_packages, monkeypatch):
        """
        Test installing a git package from a commit.
    
        This ensures Spack appropriately associates commit versions with their
        packages in time to do version lookups. Details of version lookup tested elsewhere
        """
        repo_path, filename, commits = mock_git_version_info
        monkeypatch.setattr(spack.package.PackageBase,
                            'git', 'file://%s' % repo_path,
                            raising=False)
    
        commit = commits[-1]
>       spec = spack.spec.Spec('git-test-commit@%s' % commit)

lib/spack/spack/test/cmd/install.py:254: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
lib/spack/spack/spec.py:1121: in __init__
    spec_list = SpecParser(self).parse(spec_like)
lib/spack/spack/parse.py:153: in parse
    return self.do_parse()
lib/spack/spack/spec.py:4719: in do_parse
    spec.version.generate_commit_lookup(pkg)
lib/spack/spack/version.py:492: in generate_commit_lookup
    self.commit_lookup.get(self.string)
lib/spack/spack/version.py:1057: in get
    self.data[commit] = self.lookup_commit(commit)
lib/spack/spack/version.py:1085: in lookup_commit
    self.fetcher.git("fetch", '--tags')
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <exe: ['/usr/bin/git', '-c', 'advice.detachedHead=false']>, args = ('fetch', '--tags'), kwargs = {}, env_arg = None
env = {'COLORTERM': 'truecolor', 'DBUS_SESSION_BUS_ADDRESS': 'unix:path=/run/user/1000/bus', 'DEBEMAIL': 'me@harmenstoppels.nl', 'DEBFULLNAME': 'Harmen Stoppels', ...}
EnvironmentModifications = <class 'spack.util.environment.EnvironmentModifications'>, extra_env = {}, fail_on_error = True, ignore_errors = (), ignore_quotes = False
input = None, output = None, error = None

    def __call__(self, *args, **kwargs):
        """Run this executable in a subprocess.
    
            Parameters:
                *args (str): Command-line arguments to the executable to run
    
            Keyword Arguments:
                _dump_env (dict): Dict to be set to the environment actually
                    used (envisaged for testing purposes only)
                env (dict or EnvironmentModifications): The environment with which
                    to run the executable
                extra_env (dict or EnvironmentModifications): Extra items to add to
                    the environment (neither requires nor precludes env)
                fail_on_error (bool): Raise an exception if the subprocess returns
                    an error. Default is True. The return code is available as
                    ``exe.returncode``
                ignore_errors (int or list): A list of error codes to ignore.
                    If these codes are returned, this process will not raise
                    an exception even if ``fail_on_error`` is set to ``True``
                ignore_quotes (bool): If False, warn users that quotes are not needed
                    as Spack does not use a shell. Defaults to False.
                input: Where to read stdin from
                output: Where to send stdout
                error: Where to send stderr
    
            Accepted values for input, output, and error:
    
            * python streams, e.g. open Python file objects, or ``os.devnull``
            * filenames, which will be automatically opened for writing
            * ``str``, as in the Python string type. If you set these to ``str``,
              output and error will be written to pipes and returned as a string.
              If both ``output`` and ``error`` are set to ``str``, then one string
              is returned containing output concatenated with error. Not valid
              for ``input``
            * ``str.split``, as in the ``split`` method of the Python string type.
              Behaves the same as ``str``, except that value is also written to
              ``stdout`` or ``stderr``.
    
            By default, the subprocess inherits the parent's file descriptors.
    
            """
        # Environment
        env_arg = kwargs.get('env', None)
    
        # Setup default environment
        env = os.environ.copy() if env_arg is None else {}
        self.default_envmod.apply_modifications(env)
        env.update(self.default_env)
    
        from spack.util.environment import EnvironmentModifications  # no cycle
    
        # Apply env argument
        if isinstance(env_arg, EnvironmentModifications):
            env_arg.apply_modifications(env)
        elif env_arg:
            env.update(env_arg)
    
        # Apply extra env
        extra_env = kwargs.get('extra_env', {})
        if isinstance(extra_env, EnvironmentModifications):
            extra_env.apply_modifications(env)
        else:
            env.update(extra_env)
    
        if '_dump_env' in kwargs:
            kwargs['_dump_env'].clear()
            kwargs['_dump_env'].update(env)
    
        fail_on_error = kwargs.pop('fail_on_error', True)
        ignore_errors = kwargs.pop('ignore_errors', ())
        ignore_quotes = kwargs.pop('ignore_quotes', False)
    
        # If they just want to ignore one error code, make it a tuple.
        if isinstance(ignore_errors, int):
            ignore_errors = (ignore_errors, )
    
        input  = kwargs.pop('input',  None)
        output = kwargs.pop('output', None)
        error  = kwargs.pop('error',  None)
    
        if input is str:
            raise ValueError('Cannot use `str` as input stream.')
    
        def streamify(arg, mode):
            if isinstance(arg, string_types):
                return open(arg, mode), True
            elif arg in (str, str.split):
                return subprocess.PIPE, False
            else:
                return arg, False
    
        ostream, close_ostream = streamify(output, 'w')
        estream, close_estream = streamify(error,  'w')
        istream, close_istream = streamify(input,  'r')
    
        if not ignore_quotes:
            quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
            if quoted_args:
                tty.warn(
                    "Quotes in command arguments can confuse scripts like"
                    " configure.",
                    "The following arguments may cause problems when executed:",
                    str("\n".join(["    " + arg for arg in quoted_args])),
                    "Quotes aren't needed because spack doesn't use a shell. "
                    "Consider removing them.",
                    "If multiple levels of quotation are required, use "
                    "`ignore_quotes=True`.")
    
        cmd = self.exe + list(args)
    
        escaped_cmd = ["'%s'" % arg.replace("'", "'\"'\"'") for arg in cmd]
        cmd_line_string = " ".join(escaped_cmd)
        tty.debug(cmd_line_string)
    
        try:
            proc = subprocess.Popen(
                cmd,
                stdin=istream,
                stderr=estream,
                stdout=ostream,
                env=env)
            out, err = proc.communicate()
    
            result = None
            if output in (str, str.split) or error in (str, str.split):
                result = ''
                if output in (str, str.split):
                    outstr = text_type(out.decode('utf-8'))
                    result += outstr
                    if output is str.split:
                        sys.stdout.write(outstr)
                if error in (str, str.split):
                    errstr = text_type(err.decode('utf-8'))
                    result += errstr
                    if error is str.split:
                        sys.stderr.write(errstr)
    
            rc = self.returncode = proc.returncode
            if fail_on_error and rc != 0 and (rc not in ignore_errors):
                long_msg = cmd_line_string
                if result:
                    # If the output is not captured in the result, it will have
                    # been stored either in the specified files (e.g. if
                    # 'output' specifies a file) or written to the parent's
                    # stdout/stderr (e.g. if 'output' is not specified)
                    long_msg += '\n' + result
    
>               raise ProcessError('Command exited with status %d:' %
                                   proc.returncode, long_msg)
E                                  spack.util.executable.ProcessError: Command exited with status 1:
E                                      '/usr/bin/git' '-c' 'advice.detachedHead=false' 'fetch' '--tags'

lib/spack/spack/util/executable.py:225: ProcessError
-------------------------------------------------------------------------- Captured stdout setup --------------------------------------------------------------------------
Initialized empty Git repository in /tmp/pytest-of-harmen/pytest-23/test_install_commit0/git_repo/.git/
[master (root-commit) 39234d4] first commit
 Date: Thu Oct 1 12:00:00 2020 +0200
 1 file changed, 1 insertion(+)
 create mode 100644 file.txt
[master c070216] second commit
 Date: Wed Jan 1 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 5531b94] first 1.x commit
 Date: Thu Jan 2 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x ace0d39] second 1.x commit
 Date: Fri Jan 3 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master 58eb58e] third main commit
 Date: Sat Jan 4 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[master aff22cd] fourth main commit
 Date: Sun Jan 5 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 7772e5a] third 1.x commit
 Date: Mon Jan 6 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
[1.x 27b83d1] fourth 1.x commit
 Date: Tue Jan 7 12:00:00 2020 +0300
 1 file changed, 1 insertion(+), 1 deletion(-)
-------------------------------------------------------------------------- Captured stderr setup --------------------------------------------------------------------------
Switched to a new branch '1.x'
Switched to branch 'master'
Switched to branch '1.x'
-------------------------------------------------------------------------- Captured stderr call ---------------------------------------------------------------------------
warning: no common commits
From file:///tmp/pytest-of-harmen/pytest-23/test_install_commit0/git_repo
 * branch            HEAD       -> FETCH_HEAD
 ! [rejected]        1.2        -> 1.2  (would clobber existing tag)
 ! [rejected]        v1.0       -> v1.0  (would clobber existing tag)
 ! [rejected]        v1.1       -> v1.1  (would clobber existing tag)
 ! [rejected]        v2.0       -> v2.0  (would clobber existing tag)

hitting this

[haampie]

ping @becker33, any clue?

[alalazo]

It seems the same as one of the two reported in the description.

[alalazo]

It's been a while since this came up, and git versions have been modified in the meanwhile.