Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make clear every test's status in every CI run #1679

Merged
merged 32 commits into from
Sep 26, 2023

Commits on Sep 25, 2023

  1. Instrument workflows to investigate skipped tests

    This makes jobs from both test workflows give more information
    relevant to examining which tests are skipped (and if any tests
    xfail, those too) in what environments:
    
    - Values of os.name and git.util.is_win.
    
    - The name of each test that runs, with its status.
    
    The latter doesn't increase the output length as much as might be
    expected, because due to the way the output is handled, the
    pytest-sugar pretty output format without -v looked like:
    
     test/test_actor.py ✓                                              0%
     test/test_actor.py ✓✓                                             0% ▏
     test/test_actor.py ✓✓✓                                            1% ▏
     test/test_actor.py ✓✓✓✓                                           1% ▏
    
    When instead it was intended to fit on a single line. Still, the
    current output with -v has extra newlines, increasing length and
    worsening readability, so it should be improved on if possible.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    45773c2 View commit details
    Browse the repository at this point in the history
  2. Show version and platform info in one place

    Instead of splitting it in into two places where at least one of
    the places is highly likely to be missed, this puts it together
    just before the first steps that makes nontrivial use of the
    installed packages. Grouping it together, it can't really be shown
    earlier, because one of the pieces of information is obtained
    using the git module (to examine that behavior of the code).
    
    This also presents the information more clearly. "set -x" makes
    this easy, so the commands are rewritten to take advantage of it.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    6fbe511 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    bd3307a View commit details
    Browse the repository at this point in the history
  4. Show all the failures

    Don't stop after the first 10.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    680d795 View commit details
    Browse the repository at this point in the history
  5. Keep sugar for local use, but use instafail on CI

    There are two benefits of the pytest-sugar plugin:
    
    1. Pretty output.
    2. Show details on each failure immediately instead of at the end.
    
    The first benefit is effectively local-only, because extra newlines
    are appearing when it runs on CI, both with and without -v.
    
    The second benefit applies both locally and on CI.
    
    So this adds the pytest-instafail plugin and uses it on CI to get
    the second benefit. It is not set up to run automatically, and
    pytest-sugar still is (though no longer forced), so local testing
    retains no benefit and we don't have a clash.
    
    The name "instafail" refers only to instantly *seeing* failures:
    it does not cause the pytest runner to stop earlier than otherwise.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    75cf540 View commit details
    Browse the repository at this point in the history
  6. Pass -v twice to see full skip reasons

    + Reorder pytest arguments so both workflows are consistent.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    eb56e7b View commit details
    Browse the repository at this point in the history
  7. Force pytest color output on CI

    While pytest-sugar output gets mangled with extra newlines on CI,
    colorized output seems to work fine and improves readability.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    9c7ff1e View commit details
    Browse the repository at this point in the history
  8. Fix test_blocking_lock_file for cygwin

    This permits the longer delay in test_blocking_lock_file--which was
    already allowed for native Windows--on Cygwin, where it is also
    needed. That lets the xfail mark for Cygwin be removed.
    
    This also updates the comments to avoid implying that the need for
    the delay is AppVeyor-specific (it seems needed on CI and locally).
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    0eb38bc View commit details
    Browse the repository at this point in the history
  9. Run cygpath tests on Cygwin, not native Windows

    They were not running on Cygwin, because git.util.is_win is False
    on Cygwin. They were running on native Windows, with a number of
    them always failing; these failures had sometimes been obscured by
    the --maxfail=10 that had formerly been used (from pyproject.toml).
    
    Many of them (not all the same ones) fail on Cygwin, and it might
    be valuable for cygpath to work on other platforms, especially
    native Windows. But I think it still makes sense to limit the tests
    to Cygwin at this time, because all the uses of cygpath in the
    project are in code that only runs after a check that the platform
    is Cygwin. Part of that check, as it is implemented, explicitly
    excludes native Windows (is_win must be false).
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    715dba4 View commit details
    Browse the repository at this point in the history
  10. Mark some cygpath tests xfail

    Two of the groups of cygpath tests in test_util.py generate tests
    that fail on Cygwin. There is no easy way to still run, but xfail,
    just the specific tests that fail, because the groups of tests are
    generated with `@ddt` parameterization, but neither the unittest
    nor pytest xfail mechanisms interact with that. If
    `@pytest.mark.parametrized` were used, this could be done. But
    that does not work on methods of test classes that derive from
    unittest.TestCase, including those in this project that indirectly
    derive from it by deriving from TestBase. The TestBase base class
    cannot be removed without overhauling many tests, due to fixtures
    it provides such as rorepo.
    
    So this marks too many tests as xfail, but in doing so allows test
    runs to pass while still exercising and showing status on all the
    tests, allowing result changes to be observed easily.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    d6a2d28 View commit details
    Browse the repository at this point in the history
  11. Run test_commit_msg_hook_success on more systems

    This changes a default Windows skip of test_commit_msg_hook_success
    to an xfail, and makes it more specific, expecting failure only
    when either bash.exe is unavailable (definitely expected) or when
    bash.exe is the WSL bash wrapper in System32, which fails for some
    reason even though it's not at all clear it ought to.
    
    This showcases the failures rather than skipping, and also lets the
    test pass on Windows systems where bash.exe is something else,
    including the Git Bash bash.exe that native Windows CI would use.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    881456b View commit details
    Browse the repository at this point in the history
  12. No longer skip test_index_mutation on Cygwin

    As it seems to be working now on Cygwin (maybe not native Windows).
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    c6a586a View commit details
    Browse the repository at this point in the history
  13. Report encoding error in test_add_unicode as error

    This makes the test explicitly error out, rather than skipping, if
    it appears the environment doesn't support encoding Unicode
    filenames. Platforms these days should be capable of that, and
    reporting it as an error lessens the risk of missing a bug in the
    test code (that method or a fixture) if one is ever introduced.
    Erroring out will also make it easier to see the details in the
    chained UnicodeDecodeError exception.
    
    This does not affect the behavior of GitPython itself. It only
    changes how a test reports an unusual condition that keeps the test\
    from being usefully run.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    fc02230 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    203da23 View commit details
    Browse the repository at this point in the history
  15. Report <2.5.1 in test_linked_worktree_traversal as error

    Although GitPython does not require git >=2.5.1 in general, and
    this does *not* change that, this makes the unavailability of git
    2.5.1 or later an error in test_linked_worktree_traversal, where it
    is needed to exercise that test, rather than skipping the test.
    
    A few systems, such as CentOS 7, may have downstream patched
    versions of git that remain safe to use yet are numbered <2.5.1
    and do not have the necesary feature to run this test. But by now,
    users of those systems likely anticipate that other software would
    rely on the presence of features added in git 2.5.1, which was
    released over 7 years ago. As such, I think it is more useful to
    give an error for that test, so the test's inability to be run on
    the system is clear, than to automatically skip the test, which is
    likely to go unnoticed.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    cf5f1dc View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    8923236 View commit details
    Browse the repository at this point in the history
  17. Express known test_depth failure with xfail

    Rather than skipping, so it becomes known if the situation changes.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    b198bf1 View commit details
    Browse the repository at this point in the history
  18. Remove no-effect @skipIf on test_untracked_files

    It looked like test_untracked_files was sometimes skipped, and
    specifically that it would be skipped on Cygwin. But the `@skipIf`
    on it had the condition:
    
        HIDE_WINDOWS_KNOWN_ERRORS and Git.is_cygwin()
    
    HIDE_WINDOWS_KNOWN_ERRORS can only ever be true if it is set to a
    truthy value directly (not an intended use as it's a "constant"),
    or on native Windows systems: no matter how the environment
    variable related to it is set, it's only checked if is_win, which
    is set by checking os.name, which is only "nt" on native Windows
    systems, not Cygwin.
    
    So whenever HIDE_WINDOWS_KNOWN_ERRORS is true Git.is_cygwin() will
    be false. Thus this condition is never true and the test was never
    being skipped anyway: it was running and passing on Cygwin.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    cd175a5 View commit details
    Browse the repository at this point in the history
  19. Make 2 more too-low git version skips into errors

    In the tests only, and not in any way affecting the feature set or
    requirements of GitPython itself.
    
    This is similar to, and with the same reasoning as, cf5f1dc.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    f38cc00 View commit details
    Browse the repository at this point in the history
  20. Update test_root_module Windows skip reason

    The current cause of failure is different from what is documented
    in the skip reason.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    8fd56e7 View commit details
    Browse the repository at this point in the history
  21. Change test_root_module Windows skip to xfail

    And rewrite the reason to give more useful information. (The new
    reason also doesn't state the exception type, because that is now
    specified, and checked by pytest, by being passed as "raises".)
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    c1798f5 View commit details
    Browse the repository at this point in the history
  22. Update test_git_submodules_and_add_sm_with_new_commit skip reason

    This is working on Cygwin, so that old reason no longer applies.
    (The test was not being skipped on Cygwin, and was passing.)
    
    It is not working on native Windows, due to a PermissionError from
    attempting to move a file that is still open (which Windows doesn't
    allow). That may have been the original native Windows skip reason,
    but the old AppVeyor CI link for it is broken or not public. This
    makes the reason clear, though maybe I should add more details.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    ba56752 View commit details
    Browse the repository at this point in the history
  23. Change test_git_submodules_and_add_sm_with_new_commit Windows skip to…

    … xfail
    
    And improve details.
    
    The xfail is only for native Windows, not Cygwin (same as the old
    skip was, and still via checking HIDE_WINDOWS_KNOWN_ERRORS).
    
    This change is analogous to the change in c1798f5, but for
    test_git_submodules_and_add_sm_with_new_commit rather than
    test_root_module.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    8704d1b View commit details
    Browse the repository at this point in the history
  24. Run the tests in test_tree on Windows

    This stops skipping them, as they are now working.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    1d6abdc View commit details
    Browse the repository at this point in the history
  25. Add missing raises keyword for test_depth xfail

    I had forgotten to do this earlier when converting from skip to
    xfail. Besides consistency with the other uses of xfail in the test
    suite, the benefit of passing "raises" is that pytest checks that
    the failure gave the expected exception and makes it a non-expected
    failure if it didn't.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    5609faa View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    ed95e8e View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    ceb4dd3 View commit details
    Browse the repository at this point in the history
  28. Configuration menu
    Copy the full SHA
    3276aac View commit details
    Browse the repository at this point in the history
  29. Try to work in all LF on Cygwin CI

    + Style tweak and comment to clarify the "Limit $PATH" step.
    EliahKagan committed Sep 25, 2023
    Configuration menu
    Copy the full SHA
    5d40976 View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    dda4286 View commit details
    Browse the repository at this point in the history

Commits on Sep 26, 2023

  1. Remove the recently added "Limit $PATH" step

    I had put that step in the Cygwin workflow for purposes of
    experimentation, and it seemed to make clearer what is going on,
    but really it does the opposite: it's deceptive because Cygwin uses
    other logic to set its PATH. So this step is unnecessary and
    ineffective at doing what it appears to do.
    EliahKagan committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    3007abc View commit details
    Browse the repository at this point in the history
  2. Further reduce differences between test workflows

    This makes the two CI test workflows more similar in a couple of
    the remaining ways they differ unnecessarily.
    
    This could be extended, and otherwise improved upon, in the future.
    EliahKagan committed Sep 26, 2023
    Configuration menu
    Copy the full SHA
    4860f70 View commit details
    Browse the repository at this point in the history