debian upgrade to 3.0 hangs daemon due to privilege issue

[olaf7]

After upgrading to 3.x from 2.x using apt upgrade on Debian Bookworm scm-server semi works.
When you start scm-server it starts and runs but Jetty does not serve a webapp at /scm
I run it at port 8084.
This can be confirmed using lynx from localhost: lynx http://localhost:8084 want to redirect to http://localhost:8084/scm which gives a 404.
Same is true when starting the bin/scm-server from commandline as system-user 'scm'
It works fine when bin/scm-server is started as root

When I change debugging (level and/or to logging.xml) I do not get more information. Logging stops at a fixed point. See below.
Also nothing can be found in syslog

/opt/scm-server$ bin/scm-server
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
Set http request header size to 16384
Set http response header size to 16384
Set forward request customizer: true
Set http address binding to 0.0.0.0
Set http port to 8084
Set webapp war file to /opt/scm-server/var/webapp/scm-webapp.war
Set webapp temp directory to /var/cache/scm/work/webapp
Set docroot temp directory to /var/cache/scm/work/work/docroot

When running as root it continues:

/opt/scm-server# bin/scm-server
SLF4J: No SLF4J providers were found.
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See https://www.slf4j.org/codes.html#noProviders for further details.
Set http request header size to 16384
Set http response header size to 16384
Set forward request customizer: true
Set http address binding to 0.0.0.0
Set http port to 8084
Set webapp war file to /opt/scm-server/var/webapp/scm-webapp.war
Set webapp temp directory to /var/cache/scm/work/webapp
Set docroot temp directory to /var/cache/scm/work/work/docroot
23:22:58.007 [main] INFO sonia.scm.lifecycle.classloading.ClassLoaderLifeCycle - create new simple ClassLoaderLifeCycle
23:22:58.021 [main] INFO sonia.scm.lifecycle.BootstrapContextFilter - register for restart events
23:22:58.025 [main] INFO sonia.scm.event.LegmanScmEventBus - create new event bus ScmEventBus-1
23:22:58.118 [main] DEBUG com.github.legman.shiro.ShiroExecutorDecoratorFactory - register class com.github.legman.shiro.ShiroExecutorDecoratorFactory as interface com.github.legman.ExecutorDecoratorFactory for legman
23:22:58.124 [main] DEBUG io.micrometer.common.util.internal.logging.InternalLoggerFactory - Using SLF4J as the default logging framework
23:22:58.160 [main] INFO sonia.scm.event.ScmEventBus - use sonia.scm.event.LegmanScmEventBus as event bus implementation
23:22:58.244 [main] INFO sonia.scm.lifecycle.BootstrapContextListener - start scm-manager initialization
23:22:58.244 [main] DEBUG sonia.scm.lifecycle.classloading.SimpleClassLoaderLifeCycle - init classloader sonia.scm.lifecycle.classloading.BootstrapClassLoader@7203c7ff

As there is no logging debugging this is hard.

When I change the user in systemd to root it works again. So it must be a pure privilege issue. When I change back to user scm the problem persists.
I also looked for a sudden impact of apparmor although this did not happen in the past. However I did not see anything reported by apparmor.

[pfeuffer]

Hi @olaf7 ,

could you please check your permissions for /var/cache/scm/work? This directory (and all subdirectories) have to be writable by scm. By default, this should look like this:

vagrant@bullseye:~$ ls -ld /var/cache/scm/work
drwxr-xr-x 4 scm scm 4096 Feb  7 06:53 /var/cache/scm/work

When I change the owner of this directory to root, I get the 404 you are reporting, too.

Please let us know, whether this helps.

Thanks
René

[olaf7]

Hi Weird. I did not change settings and ownership during/after upgrade, but did check ownership when running into problems. Maybe cache ownership changed when running as another user. This directory does have the right ownership and rights, however its subdirectory webapp did not. It had ownership root:scm. Once changed it now works. Also the logging now works. In the past there was a log for the initial stage of start of the daemon and one for the actual usage. Now the first part is no longer logged and logging starts(?) at the point where previously the daemon hanged due to file permissions. This makes debugging harder. Is it an idea to add logging to the start of the startup process again? And/or add a test for permissions? Thank you for testing and pointing me in the right direction, Olaf

[pfeuffer]

Hi,

glad that it works now, thanks for the update. And we will definitely consider your suggestions. They can save (and would have saved) some debugging time.

Regards
René