Issues on shared system - file permission

[Sabryr]

I am setting this RoseTTAFold on shared system where the users do not have write access to instlation folder. I have installed successfully, but when I run the test I see that RoseTTAFold try to write to the central installation.

run_pyrosetta_ver.sh input.fa .
(network.stderr file)
PermissionError: [Errno 13] Permission denied: '/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache/trans_Q/mutex'

Is there a way to tell RoseTTAFold to use the user home directory instead ?

cat network.stderr 
DGL backend not selected or invalid.  Assuming PyTorch for now.
Using backend: pytorch
Traceback (most recent call last):
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/predict_pyRosetta.py", line 200, in <module>
    pred.predict(args.a3m_fn, args.out_prefix, args.hhr, args.atab)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/predict_pyRosetta.py", line 158, in predict
    logit_s, init_crds, pred_lddt = self.model(msa, seq, idx_pdb, t1d=t1d, t2d=t2d)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/RoseTTAFoldModel.py", line 53, in forward
    msa, pair, xyz, lddt = self.feat_extractor(msa, pair, seq1hot, idx)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/Attention_module_w_str.py", line 476, in forward
    msa, pair, xyz = self.iter_block_2[i_m](msa, pair, xyz, seq1hot, idx, top_k=top_ks[i_m])
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/Attention_module_w_str.py", line 363, in forward
    xyz, state = self.str2str(msa.float(), pair.float(), xyz.float(), seq1hot, idx, top_k=top_k)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 141, in decorate_autocast
    return func(*args, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/Attention_module_w_str.py", line 241, in forward
    shift = self.se3(G, msa.reshape(B*L, -1, 1), l1_feats)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1051, in _call_impl
    return forward_call(*input, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 141, in decorate_autocast
    return func(*args, **kwargs)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/SE3_network.py", line 104, in forward
    basis, r = get_basis_and_r(G, self.num_degrees-1)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/modules.py", line 102, in get_basis_and_r
    basis = get_basis(G, max_degree, compute_gradients)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/modules.py", line 61, in get_basis
    Q_J = utils_steerable._basis_transformation_Q_J(J, d_in, d_out)
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache_file.py", line 75, in wrapper
    with FileSystemMutex(mutexfile):
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache_file.py", line 42, in __enter__
    self.acquire()
  File "/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache_file.py", line 26, in acquire
    self.handle = open(self.filename, 'w')
PermissionError: [Errno 13] Permission denied: '/cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache/trans_Q/mutex'

Regards,
Sabry

[Sabryr]

RoseTTAFold/network/equivariant_attention/from_se3cnn/cache_file.py

Line 21 in d0de739

def acquire(self):

21 def acquire(self):
22 '''
23 Locks the mutex
24 if it is already locked, it waits (blocking function)
25 '''
26 self.handle = open(self.filename, 'w')
27 fcntl.lockf(self.handle, fcntl.LOCK_EX)
28 self.handle.write("{}\n".format(os.getpid()))
29 self.handle.flush()

Above is the place where the block function try to write lock.

[minkbaek]

The other way is running the script at your working directory rather than the script directory.

For example,

1. copying input.fa under your working directory (e.g. /home/user/test_example)
2. change directory to the working dir
   cd /home/user/test_example
3. running the script
   /cluster/software/RoseTTAFold/1.0.0.1/RoseTTAFold/run_pyrosetta_ver.sh input.fa .

The current script requires writing permission for the directory where you're running the script. If you want to install RoseTTAFold for all system users and don't want to give write permission for all users, please let users run their jobs on their home directory.

[Sabryr]

Thank you for the reply. Yes the input.fa and the out is written to a folder in the home directory. Then the analysis go as far as hhsearch
started
Running HHblits
Running PSIPRED
Running hhsearch

[ec-sabryr@login-1 example]$ pwd
/home/sabryr/RoseTTAFold/RoseTTAFold/example

[ec-sabryr@login-1 example]$ ls
hhblits log other slurm-6139.out t000_.hhr t000_.msa0.ss2.a3m
input.fa network.stderr RoseTTAFold.slurm t000_.atab t000_.msa0.a3m t000_.ss2

Why the program is trying to write to the following location ?
(RoseTTAFold _DIR)/network/equivariant_attention/from_se3cnn/cache/trans_Q/mutex

It seems some lock with the PID to stop multiple processing.

[minkbaek]

I just updated the script (RoseTTAFold _DIR)/network/equivariant_attention/from_se3cnn/cache_file.py to store PID information at the current working directory rather than RoseTTAFold script directory. Please pull the latest version and try it again.

[Sabryr]

Thank you, Trying now.

[Sabryr]

I see that you have changed to " mutexfile = "cache_mutex" . I will run this now. However, in a shared system this would be ideal in a cache directory under home $HOME/.RoseTTAFold for example.

[Sabryr]

With commit #289b224a08d7883082ca25939c7398d34056efbf it worked on the shared system. Thank you.

[james-vincent]

It seems this problem still exists when running the script run_e2e_ver.sh:

Reading CRF from K4000.crf ...
Adding cs-pseudocounts ...
Effective number of sequences exp(entropy) = 11.30
Wrote profile with 779 columns as checkpoint file to t000_.msa0.tmp.chk
Using backend: pytorch
/programs/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/predict_e2e.py:161: DeprecationWarning: np.bool is a deprecated alias for the builtin bool. To silence this warning, use bool by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use np.bool_ here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
sel = np.zeros((L)).astype(np.bool)
Traceback (most recent call last):
File "/programs/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/predict_e2e.py", line 324, in
pred.predict(args.a3m_fn, args.out_prefix, args.hhr, args.atab)
File "/programs/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/predict_e2e.py", line 178, in predict
logit_s, node, init_crds, pred_lddt = self.model(input_msa, input_seq, input_idx, t1d=input_t1d, t2d=input_t2d, return_raw=True)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/RoseTTAFoldModel.py", line 112, in forward
msa, pair, xyz, lddt = self.feat_extractor(msa, pair, seq1hot, idx)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/Attention_module_w_str.py", line 476, in forward
msa, pair, xyz = self.iter_block_2[i_m](msa, pair, xyz, seq1hot, idx, top_k=top_ks[i_m])
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/Attention_module_w_str.py", line 363, in forward
xyz, state = self.str2str(msa.float(), pair.float(), xyz.float(), seq1hot, idx, top_k=top_k)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 139, in decorate_autocast
return func(args, **kwargs)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/Attention_module_w_str.py", line 241, in forward
shift = self.se3(G, msa.reshape(BL, -1, 1), l1_feats)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/site-packages/torch/cuda/amp/autocast_mode.py", line 139, in decorate_autocast
return func(*args, **kwargs)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/SE3_network.py", line 104, in forward
basis, r = get_basis_and_r(G, self.num_degrees-1)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/equivariant_attention/modules.py", line 102, in get_basis_and_r
basis = get_basis(G, max_degree, compute_gradients)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/equivariant_attention/modules.py", line 61, in get_basis
Q_J = utils_steerable._basis_transformation_Q_J(J, d_in, d_out)
File "/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache_file.py", line 68, in wrapper
os.makedirs(dirname)
File "/programs/x86_64-linux/rosettafold/1.1.0/mamba/envs/RoseTTAFold/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/nfs/programs-dev/x86_64-linux/rosettafold/1.1.0/RoseTTAFold/network/equivariant_attention/from_se3cnn/cache/trans_Q'

From here:

cache_file.py:def cached_dirpklgz(dirname, maxsize=128): utils_steerable.py:from equivariant_attention.from_se3cnn.cache_file import cached_dirpklgz utils_steerable.py:@cached_dirpklgz("%s/cache/trans_Q"%os.path.dirname(os.path.realpath(__file__)))