Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[bug] list_test tests fail non-deterministically #13583

Closed
Apteryks opened this issue Mar 31, 2023 · 2 comments
Closed

[bug] list_test tests fail non-deterministically #13583

Apteryks opened this issue Mar 31, 2023 · 2 comments

Comments

@Apteryks
Copy link

Environment details

  • Operating System+version: Guix System
  • Compiler+version: 10.3.0
  • Conan version: 2.0.2
  • Python version: 3.9.9

Steps to reproduce

I'm using the following user conftest file:

tools_locations = {
    'apt_get': {'disabled': True},
    'bazel': {'disabled': True},
    'cmake': {'default': '3.25',
              '3.15': {'disabled': True},
              '3.16': {'disabled': True},
              '3.17': {'disabled': True},
              '3.19': {'disabled': True},
              '3.25': {}},
    'pkg_config': {'exe': 'pkg-config',
                   'default': '0.29',
                   '0.29': {}},
    'svn': {'disabled': True}}

And building using the latest version with Guix, with the following environment:

dependencies: autoconf-wrapper@2.69 automake@1.16.3 cmake@3.25.1 coreutils@8.32 git-minimal@2.39.2
+ libtool@2.4.6 meson@0.60.3 ninja@1.10.2 pkg-config@0.29.2 python-bottle@0.12.23 python-colorama@0.4.4
+ python-dateutil@2.8.2 python-distro@1.6.0 python-fasteners@0.15 python-future@0.18.2 python-jinja2@3.1.1
+ python-mock@3.0.5 python-node-semver@0.8.1 python-parameterized@0.7.4 python-patch-ng@1.17.4
+ python-pluginbase@1.0.1 python-pygments@2.12.0 python-pyjwt@2.6.0 python-pytest-xdist@2.1.0
+ python-pytest@6.2.5 python-pyyaml@5.4.1 python-requests@2.28.1 python-six@1.16.0 python-tqdm@4.64.1
+ python-urllib3@1.26.9 python-webtest@2.0.35 which@2.21

On some machines, some list_test tests fail when comparing the command output, as shown in the logs below.

Logs

conans/test/utils/tools.py:563: Exception
____________________ TestListPrefs.test_list_pkg_ids[True] _____________________
[gw13] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python

self = <conans.test.integration.command_v2.list_test.TestListPrefs object at 0x7fffee093e20>
client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
remote = True

    @pytest.mark.parametrize("remote", [True, False])
    def test_list_pkg_ids(self, client, remote):
        pattern = "zli/1.0.0:*"
        expected = textwrap.dedent(f"""\
          zli
            zli/1.0.0
              revisions
                b58eeddfe2fd25ac3a105f72836b3360 (10-11-2023 10:13:13)
                  packages
                    9a4eb3c8701508aa9458b1a73d0633783ecc2270
                      info
                        settings
                          os: Linux
                    ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
                      info
                        settings
                          os: Windows
          """)
>       self.check(client, pattern, remote, expected)

conans/test/integration/command_v2/list_test.py:296: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
pattern = 'zli/1.0.0:*', remote = True
expected = '  zli\n    zli/1.0.0\n      revisions\n        b58eeddfe2fd25ac3a105f72836b3360 (10-11-2023 10:13:13)\n          pack...bec3dc6d7f6b907b3ada0c3d3cdc83613a2b715\n              info\n                settings\n                  os: Windows\n'

    @staticmethod
    def check(client, pattern, remote, expected):
        r = "-r=default" if remote else ""
        r_msg = "default" if remote else "Local Cache"
        client.run(f"list {pattern} {r}")
        expected = textwrap.indent(expected, "  ")
        expected_output = f"{r_msg}\n" + expected
        expected_output = re.sub(r"\(.*\)", "", expected_output)
        output = re.sub(r"\(.*\)", "", str(client.out))
>       assert expected_output == output
E       AssertionError: assert 'default\n  z...os: Windows\n' == 'default\n  z...  os: Linux\n'
E         Skipping 107 identical leading characters in diff, use -v to show
E         +           9a4eb3c8701508aa9458b1a73d0633783ecc2270
E         +               info
E         +                 settings
E         +                   os: Linux
E         -           ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
E         +             ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715...
E         
E         ...Full output truncated (9 lines hidden), use '-vv' to show

conans/test/integration/command_v2/list_test.py:267: AssertionError
_______________ TestListPrefs.test_list_pkg_ids_all_rrevs[True] ________________
[gw13] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python

self = <conans.test.integration.command_v2.list_test.TestListPrefs object at 0x7fffee702670>
client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
remote = True

    @pytest.mark.parametrize("remote", [True, False])
    def test_list_pkg_ids_all_rrevs(self, client, remote):
        pattern = "zli/1.0.0#*:*"
        expected = textwrap.dedent(f"""\
          zli
            zli/1.0.0
              revisions
                f034dc90894493961d92dd32a9ee3b78 (2023-01-10 22:19:58 UTC)
                  packages
                    da39a3ee5e6b4b0d3255bfef95601890afd80709
                      info
                b58eeddfe2fd25ac3a105f72836b3360 (2023-01-10 22:19:59 UTC)
                  packages
                    9a4eb3c8701508aa9458b1a73d0633783ecc2270
                      info
                        settings
                          os: Linux
                    ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
                      info
                        settings
                          os: Windows
           """)
>       self.check(client, pattern, remote, expected)

conans/test/integration/command_v2/list_test.py:421: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
pattern = 'zli/1.0.0#*:*', remote = True
expected = '  zli\n    zli/1.0.0\n      revisions\n        f034dc90894493961d92dd32a9ee3b78 (2023-01-10 22:19:58 UTC)\n          ...bec3dc6d7f6b907b3ada0c3d3cdc83613a2b715\n              info\n                settings\n                  os: Windows\n'

    @staticmethod
    def check(client, pattern, remote, expected):
        r = "-r=default" if remote else ""
        r_msg = "default" if remote else "Local Cache"
        client.run(f"list {pattern} {r}")
        expected = textwrap.indent(expected, "  ")
        expected_output = f"{r_msg}\n" + expected
        expected_output = re.sub(r"\(.*\)", "", expected_output)
        output = re.sub(r"\(.*\)", "", str(client.out))
>       assert expected_output == output
E       AssertionError: assert 'default\n  z...os: Windows\n' == 'default\n  z...  os: Linux\n'
E         Skipping 240 identical leading characters in diff, use -v to show
E         +           9a4eb3c8701508aa9458b1a73d0633783ecc2270
E         +               info
E         +                 settings
E         +                   os: Linux
E         -           ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
E         +             ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715...
E         
E         ...Full output truncated (9 lines hidden), use '-vv' to show

conans/test/integration/command_v2/list_test.py:267: AssertionError
__________________ TestListPrefs.test_list_latest_prevs[True] __________________
[gw13] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python

self = <conans.test.integration.command_v2.list_test.TestListPrefs object at 0x7fffee488e20>
client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
remote = True

    @pytest.mark.parametrize("remote", [True, False])
    def test_list_latest_prevs(self, client, remote):
        pattern = "zli/1.0.0:*#latest"
        expected = textwrap.dedent(f"""\
          zli
            zli/1.0.0
              revisions
                b58eeddfe2fd25ac3a105f72836b3360 (2023-01-10 22:27:34 UTC)
                  packages
                    9a4eb3c8701508aa9458b1a73d0633783ecc2270
                      revisions
                        9beff32b8c94ea0ce5a5e67dad95f525 (10-11-2023 10:13:13)
                      info
                        settings
                          os: Linux
                    ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
                      revisions
                        d9b1e9044ee265092e81db7028ae10e0 (10-11-2023 10:13:13)
                      info
                        settings
                          os: Windows
          """)
>       self.check(client, pattern, remote, expected)

conans/test/integration/command_v2/list_test.py:445: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
pattern = 'zli/1.0.0:*#latest', remote = True
expected = '  zli\n    zli/1.0.0\n      revisions\n        b58eeddfe2fd25ac3a105f72836b3360 (2023-01-10 22:27:34 UTC)\n          ...92e81db7028ae10e0 (10-11-2023 10:13:13)\n              info\n                settings\n                  os: Windows\n'

    @staticmethod
    def check(client, pattern, remote, expected):
        r = "-r=default" if remote else ""
        r_msg = "default" if remote else "Local Cache"
        client.run(f"list {pattern} {r}")
        expected = textwrap.indent(expected, "  ")
        expected_output = f"{r_msg}\n" + expected
        expected_output = re.sub(r"\(.*\)", "", expected_output)
        output = re.sub(r"\(.*\)", "", str(client.out))
>       assert expected_output == output
E       AssertionError: assert 'default\n  z...os: Windows\n' == 'default\n  z...  os: Linux\n'
E         Skipping 107 identical leading characters in diff, use -v to show
E         +           9a4eb3c8701508aa9458b1a73d0633783ecc2270
E         +               revisions
E         +                 9beff32b8c94ea0ce5a5e67dad95f525 
E         +               info
E         +                 settings
E         +                   os: Linux...
E         
E         ...Full output truncated (15 lines hidden), use '-vv' to show

conans/test/integration/command_v2/list_test.py:267: AssertionError
___________________ TestListPrefs.test_list_all_prevs[True] ____________________
[gw13] linux -- Python 3.9.9 /gnu/store/slsh0qjv5j68xda2bb6h8gsxwyi1j25a-python-wrapper-3.9.9/bin/python

self = <conans.test.integration.command_v2.list_test.TestListPrefs object at 0x7fffee4ff370>
client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
remote = True

    @pytest.mark.parametrize("remote", [True, False])
    def test_list_all_prevs(self, client, remote):
        pattern = "zli/1.0.0:*#*"
        # TODO: This is doing a package_id search, but not showing info
        expected = textwrap.dedent(f"""\
          zli
            zli/1.0.0
              revisions
                b58eeddfe2fd25ac3a105f72836b3360 (2023-01-10 22:41:09 UTC)
                  packages
                    9a4eb3c8701508aa9458b1a73d0633783ecc2270
                      revisions
                        9beff32b8c94ea0ce5a5e67dad95f525 (2023-01-10 22:41:09 UTC)
                      info
                        settings
                          os: Linux
                    ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
                      revisions
                        24532a030b4fcdfed699511f6bfe35d3 (2023-01-10 22:41:09 UTC)
                        d9b1e9044ee265092e81db7028ae10e0 (2023-01-10 22:41:10 UTC)
                      info
                        settings
                          os: Windows
          """)
>       self.check(client, pattern, remote, expected)

conans/test/integration/command_v2/list_test.py:471: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

client = <conans.test.utils.tools.TestClient object at 0x7fffee4ec1c0>
pattern = 'zli/1.0.0:*#*', remote = True
expected = '  zli\n    zli/1.0.0\n      revisions\n        b58eeddfe2fd25ac3a105f72836b3360 (2023-01-10 22:41:09 UTC)\n          ...1db7028ae10e0 (2023-01-10 22:41:10 UTC)\n              info\n                settings\n                  os: Windows\n'

    @staticmethod
    def check(client, pattern, remote, expected):
        r = "-r=default" if remote else ""
        r_msg = "default" if remote else "Local Cache"
        client.run(f"list {pattern} {r}")
        expected = textwrap.indent(expected, "  ")
        expected_output = f"{r_msg}\n" + expected
        expected_output = re.sub(r"\(.*\)", "", expected_output)
        output = re.sub(r"\(.*\)", "", str(client.out))
>       assert expected_output == output
E       AssertionError: assert 'default\n  z...os: Windows\n' == 'default\n  z...  os: Linux\n'
E         Skipping 107 identical leading characters in diff, use -v to show
E         +           9a4eb3c8701508aa9458b1a73d0633783ecc2270
E         +               revisions
E         +                 9beff32b8c94ea0ce5a5e67dad95f525 
E         +               info
E         +                 settings
E         +                   os: Linux...
E         
E         ...Full output truncated (16 lines hidden), use '-vv' to show

conans/test/integration/command_v2/list_test.py:267: AssertionError
@Apteryks Apteryks changed the title [bug] list_test tests fail non-deterministically [bug] list_pkg* tests fail non-deterministically Mar 31, 2023
@Apteryks Apteryks changed the title [bug] list_pkg* tests fail non-deterministically [bug] list_test tests fail non-deterministically Mar 31, 2023
@memsharded
Copy link
Member

Thanks for the report @Apteryks

This is super weird and unexpected:

E         -           ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715
E         +             ebec3dc6d7f6b907b3ada0c3d3cdc83613a2b715...

indicates that there could be a difference in the space/blanks.
I have reviewed that test, looking for tabs/spaces differences or something. Nothing, all seems good.
Also, obviously I cannot reproduce, so I have no idea of what to look for. Could be a glitch in that particular Python interpreter version? These tests run in Windows, Linux, Mac, several different Python versions...

@franramirez688
Copy link
Contributor

Finally, it was impossible to reproduce the issue on our side.
Let's close the issue.
Don't hesitate to reopen the issue if you have any other related problems or more information to try to reproduce it.

@franramirez688 franramirez688 closed this as not planned Won't fix, can't repro, duplicate, stale Sep 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@franramirez688 @Apteryks @memsharded