Updating DIRACOS2 to OpenSSL 3

[chrisburr]

We're about to update DIRACOS2 to OpenSSL 3. This has two known consequences:

• DIRAC v7.2 client installations will no longer work (already disucussed in Use OpenSSL 3.0.x DIRACOS2#95)
• Connecting to sites running old software can fail with:

  230223 06:31:19 360263 secgsi_ClientDoCert: could not instantiate session cipher using cipher public info from server
  [FATAL] Auth failed: Secgsi: ErrParseBuffer: could not instantiate session cipher : kXGS_cert
  

The second issue is caused by the minimum DH modulus size being increased to 512 bits in OpenSSL 3 (openssl/openssl#9437) and needs to be fixed on the site side. The error from OpenSSL itself is error:0280007E:Diffie-Hellman routines::modulus too small.

I've prepared this script[1] which can be ran against a DIRAC installation to iterates all storage elements and use an installation of XRootD+OpenSSL 3 on /cvmfs/lhcb.cern.ch to see if listing a directory works. The output will show all storage elements that fail regardless of reason. The OpenSSL 3 problem will look something like:

Liverpool_MC-DST ['/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/2718/stable/linux-64/bin/xrdfs', 'root://hepgrid11.ph.liv.ac.uk:1094', 'ls', '/dpm/ph.liv.ac.uk/home/lhcb'] Exited with 52 output was:
230223 10:06:13 370025 secgsi_ClientDoCert: could not instantiate session cipher using cipher public info from server
[FATAL] Auth failed: Secgsi: ErrParseBuffer: could not instantiate session cipher : kXGS_cert

Other issues are likely caused by misconfigurations or downtimes.

Please comment here after trying it so we know how many installations are ready for OpenSSL 3.

[1] Click to show script

import collections.abc
import subprocess
from copy import deepcopy

try:
    from DIRAC import initialize
except ImportError:
    from DIRAC.Core.Base.Script import parseCommandLine as initialize
initialize()
from DIRAC import gConfig
from DIRAC.Core.Utilities.ReturnValues import returnValueOrRaise


def update(d, u):
    for k, v in u.items():
        if isinstance(v, collections.abc.Mapping):
            d[k] = update(d.get(k, {}), v)
        else:
            d[k] = v
    return d


errors = {}
for se in returnValueOrRaise(gConfig.getSections("/Resources/StorageElements")):
    options = returnValueOrRaise(gConfig.getOptionsDictRecursively(f"/Resources/StorageElements/{se}"))
    if "BaseSE" in options:
        base = returnValueOrRaise(gConfig.getOptionsDictRecursively(f"/Resources/StorageElementBases/{options['BaseSE']}"))
        options = update(deepcopy(options), base)
    if "GFAL2_XROOT" not in options:
        continue
    options = options["GFAL2_XROOT"]
    cmd = ["/cvmfs/lhcb.cern.ch/lib/var/lib/LbEnv/2718/stable/linux-64/bin/xrdfs", f"root://{options['Host']}:{options.get('Port', 1094)}", "ls", options["Path"]]
    print("Testing", se, "with:", cmd)
    try:
        subprocess.check_output(cmd, timeout=5, stderr=subprocess.STDOUT, text=True)
    except subprocess.TimeoutExpired as e:
        errors[se] = (cmd, "Timed out")
    except subprocess.CalledProcessError as e:
        errors[se] = (cmd, f"Exited with {e.returncode} output was:\n{e.stdout}")

print("\n\nFound", len(errors), "errors")
for se, (cmd, message) in errors.items():
    print(se, cmd, message)

[chrisburr]

@hmiyake Can you look at this and check if you have any storage elements which have issues? (see the annoucment at the top of this page)

[hmiyake]

I've run your script (with small modification to adapt our AccessProtocol config), and got mostly Timed out results.
Now we are discussing the result...I will let you know once we concluded.

Naive question: does the issue affect only XRootD?

[chrisburr]

Thanks for trying.

Yes the issue only affects XRootD as it's a problem with older versions of the XRootD server software (changed in xrootd/xrootd@dd54932).

[hmiyake]

Sorry to have kept you waiting...finally we concluded that there is no XRootD servers starting to show a problem since OpenSSL 3 (some storage elements have XRootD connection issue since DIRACOS2 while they accept it at DIRACOS1, but they are obviously different issue)

So please go ahead.

[chrisburr]

Thank you for checking and confirming.

(some storage elements have XRootD connection issue since DIRACOS2 while they accept it at DIRACOS1, but they are obviously different issue)

Feel free to open issues in DIRACOS2 if you want some help with debugging these as it's possible we've seen them before.

[chaen]

The same issue was found at RAL, so we need to wait for them to fix it.

[chrisburr]

This issue was only visible internally at RAL (the external gateways used a newer XRootD version). It should be fixed in the next few of days.

[chaen]

Issue has been fixed https://ggus.eu/?mode=ticket_info&ticket_id=160700

[cserf]

Dunno if it can also affect DIRAC, but Rucio reported performance degradation of some agent (deletion) after moving to openSSL 3.0 see rucio/rucio#6106 for more details.

[chaen]

After checking, we are affected :-D

[chaen]

Followed up in https://its.cern.ch/jira/browse/DMC-1372

[chrisburr]

It's looking much better with OpenSSL 3.1.0

Click for details

OpenSSL 3.1.0 14 Mar 2023 (Library: OpenSSL 3.1.0 14 Mar 2023)
time python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 99.3 msec per loop

________________________________________________________
Executed in    5.13 secs    fish           external
   usr time    1.77 secs  581.00 micros    1.77 secs
   sys time    0.07 secs   65.00 micros    0.07 secs

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 77 msec per loop

________________________________________________________
Executed in    3.93 secs    fish           external
   usr time    1.65 secs  684.00 micros    1.65 secs
   sys time    0.09 secs   76.00 micros    0.09 secs

OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)
time python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 128 msec per loop

________________________________________________________
Executed in    6.52 secs    fish           external
   usr time    3.24 secs  548.00 micros    3.24 secs
   sys time    0.05 secs   61.00 micros    0.05 secs

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 107 msec per loop

________________________________________________________
Executed in    5.50 secs    fish           external
   usr time    3.18 secs  703.00 micros    3.18 secs
   sys time    0.05 secs   78.00 micros    0.05 secs

OpenSSL 1.1.1t  7 Feb 2023
time python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 82.2 msec per loop

________________________________________________________
Executed in    4.21 secs      fish           external
   usr time  851.34 millis    0.00 micros  851.34 millis
   sys time   74.90 millis  525.00 micros   74.37 millis

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2' "ctx=gfal2.creat_context(); ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 480 msec per loop

________________________________________________________
Executed in   24.09 secs    fish           external
   usr time    1.06 secs  651.00 micros    1.05 secs
   sys time    0.08 secs  149.00 micros    0.08 secs

[chaen]

I'd say only half as bad :-D
And what if the context is in the setup statement again ?

[chrisburr]

Or 5x faster wall-time at the cost of more a bit more CPU. (I have no idea which is more important for us in practice.)

And what if the context is in the setup statement again ?

Click for details

OpenSSL 3.1.0 14 Mar 2023 (Library: OpenSSL 3.1.0 14 Mar 2023)
time python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 25.9 msec per loop

________________________________________________________
Executed in    1.40 secs      fish           external
   usr time  813.14 millis  657.00 micros  812.48 millis
   sys time   29.87 millis   62.00 micros   29.81 millis

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 42.1 msec per loop

________________________________________________________
Executed in    2.18 secs      fish           external
   usr time  924.63 millis  750.00 micros  923.88 millis
   sys time   53.06 millis   72.00 micros   52.99 millis

OpenSSL 3.0.8 7 Feb 2023 (Library: OpenSSL 3.0.8 7 Feb 2023)
time python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 41.1 msec per loop

________________________________________________________
Executed in    2.16 secs    fish           external
   usr time    1.59 secs  495.00 micros    1.59 secs
   sys time    0.03 secs   48.00 micros    0.03 secs

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 59.1 msec per loop

________________________________________________________
Executed in    3.15 secs    fish           external
   usr time    1.76 secs    0.00 micros    1.76 secs
   sys time    0.03 secs  802.00 micros    0.03 secs

OpenSSL 1.1.1t  7 Feb 2023
time python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 16.6 msec per loop

________________________________________________________
Executed in  896.13 millis    fish           external
   usr time  369.67 millis    0.00 micros  369.67 millis
   sys time   20.91 millis  546.00 micros   20.36 millis

time DAVIX_USE_LIBCURL=1 python -m timeit -n 10 -r 5 -s 'import gfal2; ctx=gfal2.creat_context()' "ctx.stat('https://eoslhcb.cern.ch/')"
10 loops, best of 5: 262 msec per loop

________________________________________________________
Executed in   13.22 secs      fish           external
   usr time  573.95 millis    0.00 micros  573.95 millis
   sys time   49.33 millis  825.00 micros   48.51 millis