Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segmentation errors for gfal-copy's GridFTP plugin on RHEL8 #18

Open
XMol opened this issue Dec 20, 2023 · 7 comments
Open

Segmentation errors for gfal-copy's GridFTP plugin on RHEL8 #18

XMol opened this issue Dec 20, 2023 · 7 comments

Comments

@XMol
Copy link

XMol commented Dec 20, 2023

(moved from cern-fts/gfal2-util)

Hello GFAL developers,

we've installed a new VM with RHEL-8 to run our regular transfer tests using the gfal2-tools. They used to run just fine on SL-7, but now our gfal-copy commands randomly fail with Segmentation faults exclusively with the gsiftp transfer protocol.

For some reason that I'm unable to figure out, no core dump files are produced, while that should be possible (I can generate one myself).

$ grep systemd-coredump /var/log/messages | tail
Dec 19 23:16:02 gm-1-kit-e systemd-coredump[3646763]: Process 3646718 (python3) of user 1000 dumped core.
Dec 19 23:16:02 gm-1-kit-e systemd[1]: systemd-coredump@258-3646762-0.service: Succeeded.
Dec 20 01:16:02 gm-1-kit-e systemd-coredump[3658820]: Resource limits disable core dumping for process 3658734 (python3).
Dec 20 01:16:02 gm-1-kit-e systemd-coredump[3658820]: Process 3658734 (python3) of user 1000 dumped core.
Dec 20 01:16:02 gm-1-kit-e systemd[1]: systemd-coredump@259-3658819-0.service: Succeeded.
Dec 20 08:31:02 gm-1-kit-e systemd-coredump[3702089]: Resource limits disable core dumping for process 3702066 (python3).
Dec 20 08:31:02 gm-1-kit-e systemd-coredump[3702089]: Process 3702066 (python3) of user 1000 dumped core.
Dec 20 08:31:02 gm-1-kit-e systemd[1]: systemd-coredump@260-3702087-0.service: Succeeded.
Dec 20 08:39:36 gm-1-kit-e systemd-coredump[3703091]: Process 3703089 (sleep) of user 1000 dumped core.#012#012Stack trace of thread 3703089:#012#0  0x00007fe5fe35d8e8 __nanosleep (libc.so.6)#012#1  0x0000564612d25b47 rpl_nanosleep (sleep)#012#2  0x0000564612d25920 xnanosleep (sleep)#012#3  0x0000564612d22a88 main (sleep)#012#4  0x00007fe5fe29ed85 __libc_start_main (libc.so.6)#012#5  0x0000564612d22b5e _start (sleep)
Dec 20 08:39:36 gm-1-kit-e systemd[1]: systemd-coredump@261-3703090-0.service: Succeeded.

Other information you might find useful...

$ python3 --version
Python 3.6.8

$ rpm -qa gfal2*
gfal2-plugin-file-2.21.5-1.el8.x86_64
gfal2-plugin-gridftp-2.21.5-1.el8.x86_64
gfal2-2.21.5-1.el8.x86_64
gfal2-plugin-http-2.21.5-1.el8.x86_64
gfal2-plugin-srm-2.21.5-1.el8.x86_64
gfal2-plugin-xrootd-2.21.5-1.el8.x86_64
gfal2-util-scripts-1.8.0-1.el8.noarch

$ uname -a
Linux gm-1-kit-e 4.18.0-513.5.1.el8_9.x86_64 #1 SMP Fri Sep 29 05:21:10 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Do you have other hints on what we should check to find out more information for you? If you know how to ensure core dumps are actually produced, that would be worth sharing.

Best regards,
Xavier.

@mpatrascoiu mpatrascoiu changed the title Segmentation errors for gfal-copy's gridftp plugin Segmentation errors for gfal-copy's GridFTP plugin on RHEL8 Dec 20, 2023
@mpatrascoiu
Copy link
Contributor

Hello Xavier,

Repeating my message from cern-fts/gfal2-util#6 , given this is about GridFTP, it's not something we want to deeply investigate. The Gfal2 plan is to abandon GridFTP support.

Maybe I can help you get a stacktrace. Given the gfal2-util clients are shell wrappers, maybe you can try to do the copy via Python3 directly:

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, "src", "dst")

Maybe this way, you'll have better chances of producing a coredump.
Once the coredump is available, we can look whether it's something evident in Gfal2 or coming from the underlying GridFTP library. If it's in Gfal2, I'm interested whether this could affect the other plugins (e.g.: http, xrootd).

Cheers,
Mihai

@XMol
Copy link
Author

XMol commented Dec 20, 2023

Thank you for the suggestion, Mihai. I'll give it a try.

@XMol
Copy link
Author

XMol commented Dec 20, 2023

This small Python script was running for about 90 minutes without breaking due to a segmentation error...

import gfal2

ctx = gfal2.creat_context()
params = ctx.transfer_parameters()

ctx.filecopy(params, 'file:////bin/bash', 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp')
while True:
  ctx.filecopy(params, 'gsiftp://ppsgridftp-kit.gridka.de:2811//pnfs/gridka.de/dteam/disk-only/force-core-dump.tmp', 'file:////dev/null')

Anyway, our regular workflow (miraculously) produced three core dump files, of which I've attached the human readable information produced by coredumpctl (gfal2.coredumpctl.txt). If you need the actual core dump files, I can provide them. If you need me to do something specific with the core dump files, please tell me.

Best regards,
Xavier.

@mpatrascoiu
Copy link
Contributor

mpatrascoiu commented Dec 20, 2023

Hello,

It does look like a double free for the same GSI security context:

                Stack trace of thread 3717519:
                #0  0x00007f19af2bfe31 __libc_free (libc.so.6)
                #1  0x00007f19b05648ec ASN1_OBJECT_free (libcrypto.so.1.1)
                #2  0x00007f19b0575198 asn1_primitive_free.localalias.0 (libcrypto.so.1.1)
                #3  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #4  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #5  0x00007f19b0575618 asn1_template_free (libcrypto.so.1.1)
                #6  0x00007f19b05752d6 asn1_item_embed_free (libcrypto.so.1.1)
                #7  0x00007f19b0575539 ASN1_item_free (libcrypto.so.1.1)
                #8  0x00007f199917d697 globus_gsi_proxy_handle_destroy (libglobus_gsi_proxy_core.so.0)
                #9  0x00007f199ac7db8d gss_delete_sec_context (libglobus_gssapi_gsi.so.4)

Could you provide the coredump for gfal-rm -vvv -t 20 gsiftp://atlasgridftp-kit.gridka.de:2811/[..] ?

Cheers,
Mihai

@XMol
Copy link
Author

XMol commented Dec 20, 2023

Here you go
core.python3.1000.1b33bd67a03e41faa9fc307a01ea236f.3717501.1703065389000000.lz4.txt. Please remove/ignore the .txt suffix when using the file. I had to append it because GitHub doesn't allow me to upload .lz4.

@XMol
Copy link
Author

XMol commented Jan 25, 2024

Hi @mpatrascoiu,

did the coredump help you in any way?

Ciao,
Xavier.

@XMol
Copy link
Author

XMol commented Feb 28, 2024

Hi @mpatrascoiu,

the core dumps keep happening for our setup; around 10 times per day, which is less than 1 % of all tests, but still...

Of course, you are free to simply reject the issue on the basis that nobody cares about either gsiftp or RHEL-8 anymore. Simply close this issue and I'll stop bothering you, since we've implemented retries in case of segfault errors in our tests.

Kind regards,
Xavier.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants