File permission err for non-default .radical/ path in ptyshell

[CalmingBreeze]

Hi !

I'm working on a web interface wrapping saga but I have configuration problem.
I'm running my app with django on apache.
Without any conf, my home is /var/www/ , so /var/www/.radical/* raise permission error
I tried to set RADICAL_BASE and it partially worked, as well as RADICAL_SAGA_LOG_TGT in /var/www/html/<my_project>

But when I tried to initiate a shell connection I got an error about /var/www/.radical/ again.
With further tracing I've ended finding this :

radical.saga/src/radical/saga/utils/pty_shell.py

Lines 228 to 230 in ed6634f

	# we need a local dir for file staging caches. At this point we use
	# $HOME, but should make this configurable (FIXME)
	self.base = os.environ['HOME'] + '/.radical/saga/adaptors/shell/'

Is it possible to override this ? Maybe using conf file ?

Besides, is there a way to display all configs values ?

[andre-merzky]

Hi @CalmingBreeze,

Is it possible to override this ? Maybe using conf file ?

that's a great catch! We should make sure that RADICAL_BASE is also interpreted for that location, I'll take this as a bug report, and we should be able to fix this shortly.

Besides, is there a way to display all configs values ?

SAGA is indeed a bit under-documented in that respect, apologies for that. Let me try to collect what information we have, will ping back.

[CalmingBreeze]

Thank you for your quick response !

I think I've found another related issue :
I managed to connect myself to my target server through ssh, but I can't get any job done due to missing .radical directory on target (used for state sync ?) :

1633092655.502 : radical.saga.pty     : 15596 : 140527094773504 : DEBUG    : Got initial shell prompt (6) (Last login: Fri Oct  1 14:50:11 2021 from xxxxxx.xxxxx.fr

With some debug enabled I got :

1633092657.252 : radical.saga.pty     : 15596 : 140527094773504 : DEBUG    : run_sync: find /var/www/.radical/saga/adaptors/sge_job/ -type f -mtime +30 -print -delete | wc -l
1633092657.252 : radical.saga.pty     : 15596 : 140527094773504 : DEBUG    : write: [   28] [   88] (find /var/www/.radical/saga/adaptors/sge_job/ -type f -mtime +30 -print -delete | wc -l\n)
1633092657.255 : radical.saga.pty     : 15596 : 140527094773504 : DEBUG    : read : [   28] [   80] (find: ‘/var/www/.radical/saga/adaptors/sge_job/’: No such file or directory\n0\n)
1633092657.255 : radical.saga.pty     : 15596 : 140527094773504 : DEBUG    : read : [   28] [   10] (PROMPT-0->)
1633092657.256 : radical.saga.cpi     : 15596 : 140527094773504 : INFO     : Purged find: ‘/var/www/.radical/saga/adaptors/sge_job/’: No such file or directory

Correct me if I'm wrong but wasn't saga supposed to try create .radical/ in my $HOME user on the target machine ?
To be sure, I checked by connecting myself remotely with the same credentials, echo $HOME is correctly set as /auto/sshlogin

[CalmingBreeze]

Found the answer myself :
In my case, there two json conf files hardcoded to $home :

radical.saga/src/radical/saga/configs/adaptors_sgejob.json

Line 13 in f453ccc

"base_workdir" : "${HOME}/.radical/saga/adaptors/sge_job/"

radical.saga/src/radical/saga/configs/adaptors_shell_job.json

Line 18 in f453ccc

"base_workdir" : "${HOME}/.radical/saga/adaptors/shell_job/"

A configurable variable like 'RADICAL_BASE_TGT' would be nice =)

[CalmingBreeze]

Hello Andre !
Any news about this env vars ? It would be nice to use the use the official package instead of my modified one !

[andre-merzky]

Oh boy, I did drop the ball on this one, didn't I? Dammit....

Please have a look at PR #841 - can you give that branch a try and let us know how it goes? Thanks!

[CalmingBreeze]

Hello, Andre, I'm currently working from home so I won't have access to the infrastructure for another week and can't test the PR.

May I ask if you have implemented a way to specify the distant sync directory ?

For example we will work to a distant infrastructure with a scratch partition, will I be able to specify RADICAL_BASE $home for my machine, and a sync directory outside of my own on the target (on the /scratch partition for example)