Spring Boot application fails with ClassNotDefFound error

[yortch]

We encountered a similar issue to this: #406

We recently upgraded to Spring Boot 2.1.1 and the following Spring Cloud versions:

        springBootVersion = '2.1.1.RELEASE'
        springCloudServicesVersion = '2.1.0.RELEASE'
        springCloudVersion = 'Greenwich.RELEASE'

The application boots and runs locally with no issues. When we deploy the application to PCF, we are seeing the following error when navigating to the route:

2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at java.lang.Thread.run(Thread.java:748)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.valves.RemoteIpValve.invoke(RemoteIpValve.java:685)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.cloudfoundry.router.ClientCertificateMapper.doFilter(ClientCertificateMapper.java:77)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:90)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at com.common.util.RedirDomainFilter.doFilter(RedirDomainFilter.java:42)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at com.web.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:33)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at com.web.filter.RequestWrapper.<init>(RequestWrapper.java:55)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] at com.account.domain.dao.UserDAO.findByUsername(UserDAO.java:437)
2019-02-19T18:14:55.388-05:00 [APP/PROC/WEB/0] [OUT] 02-19 23:14:55 ERROR [http-nio-8080-exec-10] [] org.apache.catalina.core.ContainerBase.[Tomcat].[localhost].[/].[default] Servlet.service() for servlet [default] in context with path [] threw exception [Filter execution threw an exception] with root cause, java.lang.NoClassDefFoundError: Could not initialize class com.account.domain.User

The User class above lives in a jar inside the WEB-INF\lib directory of the WAR file. The same code works locally which has us wondering is this could be a compatibility issue with the java_buildpack. We are using version 4.16: java-buildpack-offline-cflinuxfs2-v4.16.zip. The application is also using Tomcat embed version 9.0.13. Also we had been able to boot and run the application successfully in same PCF instance when using Spring Boot 2.0.1. Please let us know if there is a known incompatibility version with the java build pack and other relevant spring boot components.

Thank you,
Jorge

[nebhale]

There is no known fundamental issue with Spring Boot 2.1 as our test applications (used with every commit) are based on 2.1.2.RELEASE. For Spring Boot applications, we don’t do anything special other than inspecting the manifest for a Main-Class entry and starting the Java process with that. Specifically, Spring Boot is the only thing that manages classpaths in a Spring Boot application.

In order to do more diagnosis I’d need the output staging the application (where it describes what version of the buildpack is being used and what it’s contributing). If that doesn’t yield something interesting, we’ll probably need to examine the application itself.

[nebhale]

/cc @sabbyanandan

[yortch]

Thanks Ben... here is the staging info:

Staging app and tracing logs...
   Downloading java_buildpack_offline...
   Downloaded java_buildpack_offline
   Cell a0740c87-d72f-4256-866c-bcec332054bf creating container for instance af91139c-13ae-4541-936c-67a78faf3d53
   Cell a0740c87-d72f-4256-866c-bcec332054bf successfully created container for instance af91139c-13ae-4541-936c-67a78faf3d53
   Downloading app package...
   Downloading build artifacts cache...
   Downloaded build artifacts cache (132B)
   Downloaded app package (154.2M)
   -----> Java Buildpack v4.16 (offline) | https://github.com/cloudfoundry/java-buildpack.git#41b8ff8
   -----> Downloading Jvmkill Agent 1.16.0_RELEASE from https://java-buildpack.cloudfoundry.org/jvmkill/trusty/x86_64/jvmkill-1.16.0_RELEASE.so (found in cache)
   -----> Downloading Open Jdk JRE 1.8.0_181 from https://java-buildpack.cloudfoundry.org/openjdk/trusty/x86_64/openjdk-1.8.0_181.tar.gz (found in cache)
          Expanding Open Jdk JRE to .java-buildpack/open_jdk_jre (1.1s)
          JVM DNS caching disabled in lieu of BOSH DNS caching
   -----> Downloading Open JDK Like Memory Calculator 3.13.0_RELEASE from https://java-buildpack.cloudfoundry.org/memory-calculator/trusty/x86_64/memory-calculator-3.13.0_RELEASE.tar.gz (found in cache)
          Loaded Classes: 35318, Threads: 250
   -----> Downloading Client Certificate Mapper 1.8.0_RELEASE from https://java-buildpack.cloudfoundry.org/client-certificate-mapper/client-certificate-mapper-1.8.0_RELEASE.jar (found in cache)
   -----> Downloading Container Customizer 2.6.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-customizer/container-customizer-2.6.0_RELEASE.jar (found in cache)
   -----> Downloading Container Security Provider 1.16.0_RELEASE from https://java-buildpack.cloudfoundry.org/container-security-provider/container-security-provider-1.16.0_RELEASE.jar (found in cache)
   -----> Downloading Spring Auto Reconfiguration 2.5.0_RELEASE from https://java-buildpack.cloudfoundry.org/auto-reconfiguration/auto-reconfiguration-2.5.0_RELEASE.jar (found in cache)
   Exit status 0
   Uploading droplet, build artifacts cache...
   Uploading droplet...
   Uploading build artifacts cache...
   Uploaded build artifacts cache (132B)
   Uploaded droplet (199.8M)
   Uploading complete
   Cell a0740c87-d72f-4256-866c-bcec332054bf stopping instance af91139c-13ae-4541-936c-67a78faf3d53
   Cell a0740c87-d72f-4256-866c-bcec332054bf destroying container for instance af91139c-13ae-4541-936c-67a78faf3d53
   Cell a0740c87-d72f-4256-866c-bcec332054bf successfully destroyed container for instance af91139c-13ae-4541-936c-67a78faf3d53
Waiting for app to start...
name:              [REMOVED]
requested state:   started
routes:            [REMOVED]
last uploaded:     Wed 20 Feb 09:11:27 CST 2019
stack:             cflinuxfs2
buildpacks:        java_buildpack_offline
type:            web
instances:       1/1
memory usage:    4096M
start command:   JAVA_OPTS="-agentpath:$PWD/.java-buildpack/open_jdk_jre/bin/jvmkill-1.16.0_RELEASE=printHeapHistogram=1 -Djava.io.tmpdir=$TMPDIR
                 -Djava.ext.dirs=$PWD/.java-buildpack/container_security_provider:$PWD/.java-buildpack/open_jdk_jre/lib/ext
                 -Djava.security.properties=$PWD/.java-buildpack/java_security/java.security $JAVA_OPTS" &&
                 CALCULATED_MEMORY=$($PWD/.java-buildpack/open_jdk_jre/bin/java-buildpack-memory-calculator-3.13.0_RELEASE -totMemory=$MEMORY_LIMIT -loadedClasses=36099
                 -poolType=metaspace -stackThreads=250 -vmOptions="$JAVA_OPTS") && echo JVM Memory Configuration: $CALCULATED_MEMORY && JAVA_OPTS="$JAVA_OPTS $CALCULATED_MEMORY"
                 && MALLOC_ARENA_MAX=2 SERVER_PORT=$PORT eval exec $PWD/.java-buildpack/open_jdk_jre/bin/java $JAVA_OPTS -cp $PWD/. org.springframework.boot.loader.WarLauncher
     state     since                  cpu      memory       disk           details
#0   running   2019-02-20T15:11:45Z   330.4%   1.4G of 4G   310.1M of 1G

[yortch]

This turned out to be an application issue that was only apparent in PCF due to a difference in the yml retrieved from spring cloud config. The error was somewhat misleading which is why we initially ruled out to be a property issue. Thank your for your time reviewing this!