Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_regrtest leaves a test_python_* directory in TEMPDIR #76433

Closed
xdegaye mannequin opened this issue Dec 8, 2017 · 14 comments
Closed

test_regrtest leaves a test_python_* directory in TEMPDIR #76433

xdegaye mannequin opened this issue Dec 8, 2017 · 14 comments
Labels
3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error

Comments

@xdegaye
Copy link
Mannequin

xdegaye mannequin commented Dec 8, 2017

BPO 32252
Nosy @vstinner, @xdegaye
PRs
  • bpo-32252: Fix faulthandler_suppress_crash_report() #4794
  • [3.6] bpo-32252: Fix faulthandler_suppress_crash_report() (GH-4794) #4795
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-06-21.09:38:28.575>
    created_at = <Date 2017-12-08.08:49:13.334>
    labels = ['3.7', 'type-bug', 'tests']
    title = 'test_regrtest  leaves a test_python_* directory in TEMPDIR'
    updated_at = <Date 2019-06-21.09:38:28.570>
    user = 'https://github.com/xdegaye'

    bugs.python.org fields:

    activity = <Date 2019-06-21.09:38:28.570>
    actor = 'vstinner'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-06-21.09:38:28.575>
    closer = 'vstinner'
    components = ['Tests']
    creation = <Date 2017-12-08.08:49:13.334>
    creator = 'xdegaye'
    dependencies = []
    files = []
    hgrepos = []
    issue_num = 32252
    keywords = ['patch']
    message_count = 14.0
    messages = ['307837', '307855', '307885', '307886', '308032', '308033', '308034', '308035', '308036', '308037', '308038', '308039', '308041', '346197']
    nosy_count = 2.0
    nosy_names = ['vstinner', 'xdegaye']
    pr_nums = ['4794', '4795']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue32252'
    versions = ['Python 3.7']

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 8, 2017

    After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

    I did not test on 3.6.

    @xdegaye xdegaye mannequin added 3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error labels Dec 8, 2017
    @vstinner
    Copy link
    Member

    vstinner commented Dec 8, 2017

    After running test_regrtest in the source tree on linux, the build/ subdirectory (i.e. test.libregrtest.main.TEMPDIR) contains a new test_python_* directory that contains a core file when the core file size is unlimited.

    Can you please explain how to reproduce the bug? (Which commands should I type?)

    What is your /proc/sys/kernel/core_pattern?

    On my Fedora 27, /proc/sys/kernel/core_pattern is "|/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e".

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 9, 2017

    Can you please explain how to reproduce the bug? (Which commands should I type?)

    ./python -m test test_regrtest

    What is your /proc/sys/kernel/core_pattern?

    $ cat /proc/sys/kernel/core_pattern
    core-%e.%s
    $ ulimit -c
    unlimited

    FWIW on archlinux (my platform) coredumps are handled by systemd and to enable coredumps the file /etc/sysctl.d/50-coredump.conf has to be updated to contain:

    kernel.core_pattern=core-%e.%s

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 9, 2017

    Oh it seems your Fedora 27 also uses systemd and uses the same configuration as archlinux, see https://wiki.archlinux.org/index.php/Core_dump. In that case according to this wiki your core dumps go to /var/lib/systemd/coredump and this may explain the different behaviors between your system and mine.

    My setup (updating the file /etc/sysctl.d/50-coredump.conf) was the documented archlinux practice 4 years ago when systemd used to store the coredumps in log files (very annoying). This setup is still working although it is not documented in this wiki (maybe somewhere else).

    FWIW I have kept the notes made when configuring archlinux at that time, they are:

    disable core dumps managed by systemd-coredumpctl(1) - see also man pages for sysctl.d and sysctl
    /etc/sysctl.d/50-coredump.conf # same file name as in /usr/lib/sysctl.d/
    kernel.core_pattern=core-%e.%s
    echo "core-%e.%s" > /proc/sys/kernel/core_pattern # or reboot

    # per user
    ulimit -c unlimited

    @vstinner
    Copy link
    Member

    I'm unable to reproduce the issue on Fedora 27:

    vstinner@apu$ cat /proc/sys/kernel/core_pattern
    core-%e.%s
    vstinner@apu$ ulimit -c
    unlimited
    vstinner@apu$ ./python -m test test_regrtest
    Run tests sequentially
    0:00:00 load avg: 0.70 [1/1] test_regrtest
    1 test OK.

    Total duration: 14 sec
    Tests result: SUCCESS

    While core dump works as expected:

    vstinner@apu$ ./python
    >>> import faulthandler
    >>> faulthandler._sigsegv()
    Segmentation fault (core dumped)
    vstinner@apu$ ls core*
    core-python.11.32456

    @vstinner
    Copy link
    Member

    Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

    vstinner@apu$ ls -d build/test_python_; ./python -m test test_regrtest -m test.test_regrtest.ArgsTestCase.test_crashed; ls -d build/test_python_
    ls: impossible d'accéder à 'build/test_python_*': No such file or directory

    Run tests sequentially
    0:00:00 load avg: 0.42 [1/1] test_regrtest
    1 test OK.

    Total duration: 537 ms
    Tests result: SUCCESS

    build/test_python_816

    So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

    • No core dump must be written, Python is supposed to disable that
    • regrtest must emit a warning if it fails to remove the temporary directory, maybe also fail with an error? (non-zero exit code)

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 11, 2017

    What is now the content of the 'build/' subdirectory of the source tree ?

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 11, 2017

    Ah! I misunderstood the bug report. I was looking for a ENV_FAILED failure, but no, regrtest fails to remove its temporary directory but no warning is emitted in this case.

    Yes :-)

    @xdegaye
    Copy link
    Mannequin Author

    xdegaye mannequin commented Dec 11, 2017

    So running test_regrtest creates build/test_python_816/ but fails to remove it because it contains a core dump: core-python.11.816.

    Not sure that it is because it contains a core file. Maybe it just does not remove it because the test crashed ?

    @vstinner
    Copy link
    Member

    Maybe it just does not remove it because the test crashed ?

    Ah, you're right :-)

    @vstinner
    Copy link
    Member

    • No core dump must be written, Python is supposed to disable that

    My bad. It's an obvious bug that was missed before because regrtest creates a temporary directory to run tests and then remove it.

    I wrote the PR 4794 to fix faulthandler.

    @vstinner
    Copy link
    Member

    New changeset 48d4dd9 by Victor Stinner in branch 'master':
    bpo-32252: Fix faulthandler_suppress_crash_report() (bpo-4794)
    48d4dd9

    @vstinner
    Copy link
    Member

    New changeset 71d8f36 by Victor Stinner (Miss Islington (bot)) in branch '3.6':
    bpo-32252: Fix faulthandler_suppress_crash_report() (GH-4794) (bpo-4795)
    71d8f36

    @vstinner
    Copy link
    Member

    This issue has been fixed in a more generic way: when using -jN, child processes reuse the temporary directory of the main process, and then main process ensures that the temporary directory is always removed:

    commit 3c93153
    Author: Victor Stinner <vstinner@redhat.com>
    Date: Tue May 14 15:49:16 2019 +0200

    bpo-36915: regrtest always remove tempdir of worker processes (GH-13312)
    

    I close the issue. Moreover, the initial issue has been fixed.

    Thanks for the report Xavier :-)

    @ezio-melotti ezio-melotti transferred this issue from another repository Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life tests Tests in the Lib/test dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    1 participant