error from `scram build`: `AttributeError: 'NoneType' object has no attribute 'st_mode'`

[missirol]

In the last few weeks, I encountered the error in [*] a few times when running scram build in a local CMSSW area on lxplus8 nodes.

Some more info.

• I'm using CMSSW_15_0_17.
• I'm not doing anything special in my area: I have a few subpackages checked out from CMSSW (git cms-addpkg ..), plus a couple of extra subpackages with my user code.
• I do not understand what triggers the issue: everything works normally for a few days (compiling, running tests, etc), then out of nowhere I get the error in [*] when trying to compile.
• I haven't found a workaround (I see the same behavior even if I use a different lxplus8 node, and scram build clean shows the same error). In other words, once I get [*], I cannot recover the situation, and I just end up building another CMSSW area from scratch.
• I do not have a recipe to reproduce the problem from scratch (but now I have a couple of areas with this problem, and maybe those could be used for diagnostics).

Questions.

• Any idea what the cause of this issue is ?
• Even if the problem itself is not caused by SCRAM, maybe it would be useful to add some extra check in SCRAM to make the error message more useful ?

---

Edit : see #83 (comment) for more insight on the cause of this problem.

---

[*]

Traceback (most recent call last):
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/bin/scram.py", line 114, in <module>
    main()
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/bin/scram.py", line 109, in main
    if not execcommand(args, opts):
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/bin/scram.py", line 103, in execcommand
    return eval('scram_commands.cmd_%s' % cmds[0])(args, opts)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/CMD.py", line 28, in cmd_build
    return process(args, opts)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 411, in process
    dircache.checkfiles(opts.reset)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 91, in checkfiles
    self.checktree(environ['SCRAM_SOURCEDIR'], True)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 164, in checktree
    self.checktree(sdir, required)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 164, in checktree
    self.checktree(sdir, required)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 164, in checktree
    self.checktree(sdir, required)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 121, in checktree
    curdirs = self.getdir(path)
  File "/cvmfs/cms.cern.ch/share/lcg/SCRAMV1/V3_00_84/SCRAM/Core/Commands/build.py", line 173, in getdir
    if S_ISDIR(st.st_mode) and (not self.buildclass(d)[2]):
AttributeError: 'NoneType' object has no attribute 'st_mode'

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @missirol.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[smuzaffar]

@missirol , could it be that your cmssw dev area is on AFS and you use the same sessions for days and after few days your AFS token is expired and you can not read your own files?

If the area is on AFS then once you get this error then try running scram build from a fresh shell. Do you still get the same error in new shell?

[smuzaffar]

Even if the problem itself is not caused by SCRAM, maybe it would be useful to add some extra check in SCRAM to make the error message more useful ?

sure, I will update scram to print out the correct message in this case

[missirol]

Hi @smuzaffar , thanks for having a look.

could it be that your cmssw dev area is on AFS and you use the same sessions for days and after few days your AFS token is expired and you can not read your own files?

I don't think this is the issue.

• My dev areas are indeed on AFS.
• What I have seen is that if my session lasts more than ~24h, I do see the problem of not being able to read my own files (e.g. if I do ls ., I get something like "Permission denied"). This is general, not related to scram. When this happens, I normally notice the issue before even trying to use scram, and I just have to log out and log back in (re-doing 2FA).

If the area is on AFS then once you get this error then try running scram build from a fresh shell. Do you still get the same error in new shell?

Yes. I did as follows.

• I logged out of lxplus8.
• I logged back in on a different lxplus8 node, after redoing 2FA.
• I went to one of the "broken" areas, did cmsenv (no problems), and if I run scram build I see the same error as per description.

[missirol]

Okay, I think I know what is triggering the issue (should have realised this earlier, sorry..).

The problem seems to be the presence of an invalid sym-link without a dot in its name.

Below is a reproducer using the most recent CMSSW pre-release.

(In my case, I had a sym-link pointing to a directory with ROOT files, and I triggered the problem by deleting that directory while forgetting to also delete the sym-link..)

#!/bin/bash -ex

cmsrel CMSSW_16_1_0_pre1
cd CMSSW_16_1_0_pre1/src
cmsenv
scram b

# works: invalid sym-link with a dot in its name
ln -sf out.root out1.root
scram b

# works: valid sym-link without a dot in its name
mkdir tmp_dir
ln -sf tmp_dir tmp_dir_link
scram b

# breaks: invalid sym-link without a dot in its name
ln -sf no_dir no_dir_link
scram b

[smuzaffar]

thanks @missirol . I agree that if scram had failed with properly error ( e.g. file name it is trying to access) then you would have noticed this earlier. I will update scram to report the correct error

[missirol]

Thanks @smuzaffar. I let you close this issue when you prefer.

[smuzaffar]

e8b116e should fix this and scram should properly print out the error message. Once tests via cmsdist then it should be available for all CMSSW releases