New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ConcurrentModification error during Tomcat startup #5115

Closed
JasonLunn opened this Issue Mar 27, 2018 · 15 comments

Comments

Projects
None yet
2 participants
@JasonLunn

JasonLunn commented Mar 27, 2018

Environment

  • jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a Java HotSpot(TM) 64-Bit Server VM 25.162-b12 on 1.8.0_162-b12 +jit [darwin-x86_64]
  • JAVA_OPTS=-Xmx2G -Djruby.debug.fullTrace=true -Djruby.cli.debug=true -Djruby.backtrace.style=raw
  • Mac OS X, version 10.12.6
  • Tomcat 7.0.62
  • Warbler from master via git
  • Rails 5.1.5

Expected Behavior

  • Expect to be able to complete startup without a JRuby-Rack error

Actual Behavior

2018-03-27 12:45:49,902 INFO  jruby.rack:27 {player.env.name=test/embedci} [localhost-startStop-1]: An exception happened during JRuby-Rack startup
load error: pp -- java.util.ConcurrentModificationException: null
--- System
jruby 9.1.16.0 (2.3.3) 2018-02-21 8f3f95a Java HotSpot(TM) 64-Bit Server VM 25.162-b12 on 1.8.0_162-b12 +jit [darwin-x86_64]
Time: 2018-03-27 12:45:49 -0400
Server: Apache Tomcat/7.0.62
jruby.home: uri:classloader://META-INF/jruby.home

--- Context Init Parameters:
jruby.compat.version = 1.9
jruby.max.runtimes = 1
jruby.min.runtimes = 1
jruby.rack.logging = slf4j
public.root = /
rails.env = test

--- Backtrace
LoadError: load error: pp -- java.util.ConcurrentModificationException: null
                                        getStackTrace at java/lang/Thread.java:1559
                                     getBacktraceData at org/jruby/runtime/backtrace/TraceType.java:244
                                         getBacktrace at org/jruby/runtime/backtrace/TraceType.java:47
                                     prepareBacktrace at org/jruby/RubyException.java:235
                                             preRaise at org/jruby/exceptions/RaiseException.java:216
                                             preRaise at org/jruby/exceptions/RaiseException.java:183
                                               <init> at org/jruby/exceptions/RaiseException.java:111
                                    newRaiseException at org/jruby/Ruby.java:4115
                                         newLoadError at org/jruby/Ruby.java:4002
                            newLoadErrorFromThrowable at org/jruby/runtime/load/LoadService.java:924
                            tryLoadingLibraryOrScript at org/jruby/runtime/load/LoadService.java:915
                                    smartLoadInternal at org/jruby/runtime/load/LoadService.java:531
                                              require at org/jruby/runtime/load/LoadService.java:402
                                        requireCommon at org/jruby/RubyKernel.java:963
                                            require19 at org/jruby/RubyKernel.java:956
                                                 call at org/jruby/RubyKernel$INVOKER$s$1$0$require19_DBG.gen:-1
                                                 call at org/jruby/internal/runtime/methods/JavaMethod.java:637
                                                 call at org/jruby/internal/runtime/methods/AliasMethod.java:101
                                      unresolvedSuper at org/jruby/ir/runtime/IRRuntimeHelpers.java:1042
                                               zSuper at org/jruby/ir/runtime/IRRuntimeHelpers.java:1055
                  invokeSuper2:-unknown-super-target- at Users/teamcity/Applications/apache_minus_tomcat/webapps/ROOT/WEB_minus_INF/gems/gems/activesupport_minus_5_dot_1_dot_5/lib/active_support//Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292
                                     block in require at /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292
                                          yieldDirect at org/jruby/runtime/CompiledIRBlockBody.java:156
                                        yieldSpecific at org/jruby/runtime/IRBlockBody.java:80
                                        yieldSpecific at org/jruby/runtime/Block.java:134
                                        yieldSpecific at org/jruby/ir/runtime/IRRuntimeHelpers.java:421
                                        yieldSpecific at org/jruby/ir/targets/YieldSite.java:114
                                      load_dependency at /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:258
                                                 call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:103
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:163
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:163
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:170
                         invokeOther4:load_dependency at Users/teamcity/Applications/apache_minus_tomcat/webapps/ROOT/WEB_minus_INF/gems/gems/activesupport_minus_5_dot_1_dot_5/lib/active_support//Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292
                                              require at /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292
                                                 call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:103
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:163
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:202
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:318
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:155
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:315
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                       INTERPRET_ROOT at org/jruby/ir/interpreter/Interpreter.java:107
                                              execute at org/jruby/ir/interpreter/Interpreter.java:94
                                              execute at org/jruby/ir/interpreter/Interpreter.java:34
                                              execute at org/jruby/ir/IRTranslator.java:42
                                       runInterpreter at org/jruby/Ruby.java:842
                                             loadFile at org/jruby/Ruby.java:2891
                                                 load at org/jruby/runtime/load/LibrarySearcher.java:243
                                                 load at org/jruby/runtime/load/LibrarySearcher.java:34
                            tryLoadingLibraryOrScript at org/jruby/runtime/load/LoadService.java:891
                                    smartLoadInternal at org/jruby/runtime/load/LoadService.java:531
                                      autoloadRequire at org/jruby/runtime/load/LoadService.java:406
                                                 load at org/jruby/RubyKernel.java:196
                                          getConstant at org/jruby/RubyModule.java:4625
                                  getAutoloadConstant at org/jruby/RubyModule.java:4422
                                  getAutoloadConstant at org/jruby/RubyModule.java:4415
                                 resolveUndefConstant at org/jruby/RubyModule.java:3828
                        getConstantFromNoConstMissing at org/jruby/RubyModule.java:3797
                                                cache at org/jruby/ir/instructions/SearchModuleForConstInstr.java:73
                                            interpret at org/jruby/ir/instructions/SearchModuleForConstInstr.java:103
                                       processOtherOp at org/jruby/ir/interpreter/StartupInterpreterEngine.java:178
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:104
                                      INTERPRET_BLOCK at org/jruby/ir/interpreter/Interpreter.java:127
                                      commonYieldPath at org/jruby/runtime/MixedModeIRBlockBody.java:149
                                              doYield at org/jruby/runtime/IRBlockBody.java:186
                                                yield at org/jruby/runtime/BlockBody.java:116
                                                yield at org/jruby/runtime/Block.java:165
                                                  tap at org/jruby/RubyKernel.java:1765
                                                 call at org/jruby/RubyKernel$INVOKER$s$0$0$tap_DBG.gen:-1
                                                 call at org/jruby/internal/runtime/methods/JavaMethod.java:536
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
                                            interpret at org/jruby/ir/instructions/CallBase.java:429
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                      INTERPRET_BLOCK at org/jruby/ir/interpreter/Interpreter.java:127
                                      commonYieldPath at org/jruby/runtime/MixedModeIRBlockBody.java:149
                                        yieldSpecific at org/jruby/runtime/IRBlockBody.java:84
                                        yieldSpecific at org/jruby/runtime/Block.java:134
                                          synchronize at org/jruby/ext/thread/Mutex.java:158
                                                 call at org/jruby/ext/thread/Mutex$INVOKER$i$0$0$synchronize_DBG.gen:-1
                                                 call at org/jruby/internal/runtime/methods/JavaMethod.java:536
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
                                            interpret at org/jruby/ir/instructions/CallBase.java:429
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                                 call at org/jruby/internal/runtime/methods/AliasMethod.java:56
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                      INTERPRET_BLOCK at org/jruby/ir/interpreter/Interpreter.java:127
                                      commonYieldPath at org/jruby/runtime/MixedModeIRBlockBody.java:149
                                              doYield at org/jruby/runtime/IRBlockBody.java:193
                                                yield at org/jruby/runtime/BlockBody.java:125
                                        yieldNonArray at org/jruby/runtime/Block.java:169
                                           yieldUnder at org/jruby/RubyBasicObject.java:1751
                                      instance_exec19 at org/jruby/RubyBasicObject.java:1728
                                                 call at org/jruby/RubyBasicObject$INVOKER$i$0$3$instance_exec19_DBG.gen:-1
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:77
                           invokeOther2:instance_exec at Users/teamcity/Applications/apache_minus_tomcat/webapps/ROOT/WEB_minus_INF/gems/gems/railties_minus_5_dot_1_dot_5/lib/rails//Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:30
                                                  run at /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:30
                                                 call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:77
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:93
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:176
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:67
                                     invokeOther1:run at Users/teamcity/Applications/apache_minus_tomcat/webapps/ROOT/WEB_minus_INF/gems/gems/railties_minus_5_dot_1_dot_5/lib/rails//Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:59
                            block in run_initializers at /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:59
                                          yieldDirect at org/jruby/runtime/CompiledIRBlockBody.java:156
                                          yieldDirect at org/jruby/runtime/MixedModeIRBlockBody.java:123
                                                yield at org/jruby/runtime/BlockBody.java:114
                                                yield at org/jruby/runtime/Block.java:165
                                                yield at org/jruby/ir/runtime/IRRuntimeHelpers.java:417
                                                yield at org/jruby/ir/targets/YieldSite.java:87
                                  block in tsort_each at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:228
                                          yieldDirect at org/jruby/runtime/CompiledIRBlockBody.java:156
                                          yieldDirect at org/jruby/runtime/MixedModeIRBlockBody.java:123
                                                yield at org/jruby/runtime/BlockBody.java:114
                                                yield at org/jruby/runtime/Block.java:165
                                                yield at org/jruby/ir/runtime/IRRuntimeHelpers.java:417
                                                yield at org/jruby/ir/targets/YieldSite.java:87
           block in each_strongly_connected_component at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:350
                                          yieldDirect at org/jruby/runtime/CompiledIRBlockBody.java:156
                                                yield at org/jruby/runtime/BlockBody.java:114
                                                yield at org/jruby/runtime/Block.java:165
                                                yield at org/jruby/ir/runtime/IRRuntimeHelpers.java:417
                                                yield at org/jruby/ir/targets/YieldSite.java:87
               each_strongly_connected_component_from at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:431
                                                 call at org/jruby/internal/runtime/methods/CompiledIRMethod.java:77
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:93
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:77
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
  invokeOther2:each_strongly_connected_component_from at uri_3a_classloader_3a_/META_minus_INF/jruby_dot_home/lib/ruby/stdlib/uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:349
           block in each_strongly_connected_component at uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:349
                                          yieldDirect at org/jruby/runtime/CompiledIRBlockBody.java:156
                                          yieldDirect at org/jruby/runtime/MixedModeIRBlockBody.java:123
                                                yield at org/jruby/runtime/BlockBody.java:114
                                                yield at org/jruby/runtime/Block.java:165
                                                 each at org/jruby/RubyArray.java:1735
                                                 call at org/jruby/RubyArray$INVOKER$i$0$0$each_DBG.gen:-1
                                                 call at org/jruby/internal/runtime/methods/JavaMethod.java:536
                                                 call at org/jruby/internal/runtime/methods/AliasMethod.java:101
                                                 call at org/jruby/RubyMethod.java:127
                                                 call at org/jruby/RubyMethod$INVOKER$i$call_DBG.gen:-1
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
                                            interpret at org/jruby/ir/instructions/CallBase.java:429
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:109
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:95
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
                                            interpret at org/jruby/ir/instructions/CallBase.java:429
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:109
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:95
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                            interpret at org/jruby/ir/instructions/CallBase.java:432
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:109
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:95
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:278
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:79
                                             callIter at org/jruby/runtime/callsite/CachingCallSite.java:84
                                            interpret at org/jruby/ir/instructions/CallBase.java:429
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:360
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:90
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:214
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:200
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:210
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:338
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:183
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:324
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                       INTERPRET_ROOT at org/jruby/ir/interpreter/Interpreter.java:107
                                              execute at org/jruby/ir/interpreter/Interpreter.java:94
                                              execute at org/jruby/ir/interpreter/Interpreter.java:34
                                              execute at org/jruby/ir/IRTranslator.java:42
                                       runInterpreter at org/jruby/Ruby.java:842
                                             loadFile at org/jruby/Ruby.java:2891
                                                 load at org/jruby/runtime/load/LibrarySearcher.java:243
                                                 load at org/jruby/runtime/load/LibrarySearcher.java:34
                            tryLoadingLibraryOrScript at org/jruby/runtime/load/LoadService.java:891
                                    smartLoadInternal at org/jruby/runtime/load/LoadService.java:531
                                              require at org/jruby/runtime/load/LoadService.java:402
                                        requireCommon at org/jruby/RubyKernel.java:963
                                            require19 at org/jruby/RubyKernel.java:956
                                                 call at org/jruby/RubyKernel$INVOKER$s$1$0$require19_DBG.gen:-1
                                                 call at org/jruby/internal/runtime/methods/JavaMethod.java:398
                                                 call at org/jruby/internal/runtime/methods/AliasMethod.java:61
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:318
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:155
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:315
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                            interpret at org/jruby/ir/interpreter/InterpreterEngine.java:78
                                     INTERPRET_METHOD at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:144
                                                 call at org/jruby/internal/runtime/methods/MixedModeIRMethod.java:130
                                                 call at org/jruby/internal/runtime/methods/DynamicMethod.java:194
                                         cacheAndCall at org/jruby/runtime/callsite/CachingCallSite.java:298
                                                 call at org/jruby/runtime/callsite/CachingCallSite.java:127
                                          processCall at org/jruby/ir/interpreter/InterpreterEngine.java:344
                                            interpret at org/jruby/ir/interpreter/StartupInterpreterEngine.java:72
                                       INTERPRET_ROOT at org/jruby/ir/interpreter/Interpreter.java:107
                                              execute at org/jruby/ir/interpreter/Interpreter.java:94
                                              execute at org/jruby/ir/interpreter/Interpreter.java:34
                                              execute at org/jruby/ir/IRTranslator.java:42
                                        evalScriptlet at org/jruby/Ruby.java:482
                                        evalScriptlet at org/jruby/Ruby.java:458
                              createApplicationObject at org/jruby/rack/rails/RailsRackApplicationFactory.java:22
                                               create at org/jruby/rack/DefaultRackApplicationFactory.java:101
                                                 init at org/jruby/rack/DefaultRackApplicationFactory.java:445
                                       getApplication at org/jruby/rack/DefaultRackApplicationFactory.java:114
                                               doInit at org/jruby/rack/SharedRackApplicationFactory.java:34
                                                 init at org/jruby/rack/RackApplicationFactoryDecorator.java:100
                                   contextInitialized at org/jruby/rack/RackServletContextListener.java:50
                                        listenerStart at org/apache/catalina/core/StandardContext.java:5016
                                        startInternal at org/apache/catalina/core/StandardContext.java:5528
                                                start at org/apache/catalina/util/LifecycleBase.java:150
                                     addChildInternal at org/apache/catalina/core/ContainerBase.java:901
                                             addChild at org/apache/catalina/core/ContainerBase.java:877
                                             addChild at org/apache/catalina/core/StandardHost.java:652
                                            deployWAR at org/apache/catalina/startup/HostConfig.java:1095
                                                  run at org/apache/catalina/startup/HostConfig.java:1930
                                                 call at java/util/concurrent/Executors.java:511
                                                  run at java/util/concurrent/FutureTask.java:266
                                            runWorker at java/util/concurrent/ThreadPoolExecutor.java:1149
                                                  run at java/util/concurrent/ThreadPoolExecutor.java:624
                                                  run at java/lang/Thread.java:748

--- Environment Variables
API_HOST = vulcan.junlabs.com
AWS_ACCESS_KEY_ID = XYZ
AWS_REGION = us-east-1
AWS_S3_BUCKET = player-static-staging
AWS_SECRET_ACCESS_KEY = ABC
BUNDLER_ORIG_BUNDLER_ORIG_MANPATH = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLER_VERSION = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_BIN_PATH = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_GEMFILE = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/Gemfile
BUNDLER_ORIG_GEM_HOME = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems
BUNDLER_ORIG_GEM_PATH = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_MANPATH = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_PATH = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RB_USER_INSTALL = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYLIB = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYOPT = BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_VERSION = 1.16.1
BUNDLE_BIN_PATH = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/bundler-1.16.1/exe/bundle
BUNDLE_GEMFILE = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/Gemfile
BUNDLE_WITHOUT = development:test:assets
CDN_HOST = vulcan-localhost-cdn.junlabs.com
COVERAGE = true
DATABASE_URL = XYZ
GCP_PROJECT_ID = marketplace-142121
GEM_HOME = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems
GOOGLE_CLIENT_ID = ABC
GOOGLE_CLIENT_SECRET = XYZ
HOME = /Users/teamcity
JAVA_HOME = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home
JAVA_MAIN_CLASS_49875 = org.apache.catalina.startup.Bootstrap
JAVA_OPTS = -Xmx2G -Djruby.debug.fullTrace=true -Djruby.cli.debug=true -Djruby.backtrace.style=raw
JBUNDLE_SKIP = true
MONIKER = ABC
PAPERTRAIL_HOST = logs.papertrailapp.com
PAPERTRAIL_PORT = 12345
PATH = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/bin
PWD = /Users/teamcity/Applications/buildAgent/work/51de57e2559977e7
RACK_ENV = production
RAILS_ENV = test
RAILS_ROOT = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF
RUBY = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/bin/java -cp :/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/classes:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/api-common-1.4.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/auto-value-1.4.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/c3p0-0.9.1.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/commons-codec-1.3.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/commons-logging-1.1.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/error_prone_annotations-2.1.2.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/gax-1.19.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/gax-grpc-1.19.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-auth-library-credentials-0.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-auth-library-oauth2-http-0.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-cloud-core-1.20.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-cloud-core-grpc-1.20.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-cloud-pubsub-0.38.0-beta.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-http-client-1.23.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/google-http-client-jackson2-1.19.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/groovy-all-2.4.7.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-auth-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-context-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-core-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-google-cloud-pubsub-v1-0.3.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-netty-shaded-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-protobuf-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-protobuf-lite-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/grpc-stub-1.9.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/gson-2.7.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/guava-20.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/httpclient-4.0.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/httpcore-4.0.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/instrumentation-api-0.4.3.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jackson-core-2.1.3.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jline-2.11.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/joda-time-2.9.2.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jruby-core-9.1.16.0-complete.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jruby-rack-1.1.21.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jruby-stdlib-9.1.16.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/jsr305-3.0.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/logback-classic-1.1.3.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/logback-core-1.1.3.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/mysql-connector-java-5.1.38-bin.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/opencensus-api-0.10.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/opencensus-contrib-grpc-metrics-0.10.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/proto-google-cloud-pubsub-v1-0.3.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/proto-google-common-protos-1.2.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/proto-google-iam-v1-0.3.0.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/protobuf-java-3.5.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/protobuf-java-util-3.5.1.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/slf4j-api-1.7.7.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/snakeyaml-1.16.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/snakeyaml-1.17.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/sockjs-servlet-0.1.2.jar:/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib/threetenbp-1.3.3.jar org.jruby.Main
RUBYLIB = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/bundler-1.16.1/lib
RUBYOPT = -rbundler/setup
SHLVL = 1
TEAMCITY_VERSION = 2017.2.2 (build 50909)
TOMCAT_HOME = /Users/teamcity/Applications/apache-tomcat
_ = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/bin/java
__CF_USER_TEXT_ENCODING = 0x1F5:0x0:0x0
gopherProxySet = false
socksNonProxyHosts = local|*.local|169.254/16|*.169.254/16

--- RubyGems
Gem.dir: /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems
Gem.path:
/Users/teamcity/.gem/jruby/2.3.0
uri:classloader:/META-INF/jruby.home/lib/ruby/gems/shared
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems
Activated gems:
  did_you_mean-1.0.1
  bundler-1.16.1
  rake-12.3.0
  concurrent-ruby-1.0.5-java
  i18n-0.9.5
  minitest-5.10.3
  thread_safe-0.3.6-java
  tzinfo-1.2.5
  activesupport-5.1.5
  builder-3.2.3
  erubi-1.7.1
  nokogiri-1.8.2-java
  rails-dom-testing-2.0.3
  crass-1.0.3
  loofah-2.2.0
  rails-html-sanitizer-1.0.3
  actionview-5.1.5
  rack-2.0.4
  rack-test-0.8.3
  actionpack-5.1.5
  nio4r-2.3.0-java
  websocket-extensions-0.1.3
  websocket-driver-0.6.5-java
  actioncable-5.1.5
  globalid-0.4.1
  activejob-5.1.5
  mini_mime-1.0.0
  mail-2.7.0
  actionmailer-5.1.5
  activemodel-5.1.5
  arel-8.0.0
  activerecord-5.1.5
  activerecord-jdbc-adapter-51.0
  jdbc-mysql-5.1.44
  activerecord-jdbcmysql-adapter-51.0
  public_suffix-3.0.2
  addressable-2.5.2
  ar-octopus-0.9.2
  aws-sigv4-1.0.2
  jmespath-1.3.1
  aws-sdk-core-2.11.16
  aws-sdk-resources-2.11.16
  aws-sdk-2.11.16
  descendants_tracker-0.0.4
  ice_nine-0.11.2
  axiom-types-0.1.1
  htmlentities-4.3.4
  rubyzip-1.0.0
  axlsx-2.0.1
  coercible-1.0.0
  commander-1.2.4
  declarative-0.0.10
  declarative-option-0.1.0
  docile-1.3.0
  unf-0.1.4-java
  domain_name-0.5.20170404
  equalizer-0.0.11
  multipart-post-2.0.0
  faraday-0.12.2
  geoip-1.6.3
  jwt-1.5.6
  little-plugger-1.1.4
  multi_json-1.13.1
  logging-2.2.2
  memoist-0.16.0
  os-0.9.6
  signet-0.8.1
  googleauth-0.6.2
  httpclient-2.8.3
  mime-types-data-3.2016.0521
  mime-types-3.1
  uber-0.1.0
  representable-3.0.4
  retriable-3.1.1
  google-api-client-0.19.8
  google-cloud-env-1.0.1
  google-cloud-core-1.2.0
  google-cloud-bigquery-1.1.0
  hashie-3.5.7
  http-cookie-1.0.3
  multi_xml-0.6.0
  httparty-0.16.1
  jar-dependencies-0.3.12
  virtus-1.0.5
  maven-tools-1.1.6
  ruby-maven-libs-3.3.9
  ruby-maven-3.3.12
  jbundler-0.9.3
  mimemagic-0.3.2
  minitest-proveit-1.0.0
  thor-0.20.0
  jglib-1.3.0
  method_source-0.9.0
  railties-5.1.5
  jquery-rails-4.3.1
  jruby-jars-9.1.16.0
  jruby-rack-1.1.21
  json-2.1.0-java
  manifest-1.0.0
  netrc-0.11.0
  newrelic_rpm-4.8.0.341
  oauth2-1.4.0
  omniauth-1.8.1
  omniauth-oauth2-1.5.0
  omniauth-google-oauth2-0.5.3
  rack-cache-1.7.1
  sprockets-3.7.1
  sprockets-rails-3.2.1
  rails-5.1.5
  rest-client-2.0.2
  rest-client-components-1.4.0
  simplecov-html-0.10.2
  simplecov-0.16.1
  simplecov-teamcity-summary-0.1.2
  state_machines-0.5.0
  state_machines-activemodel-0.5.0
  state_machines-activerecord-0.5.0
  useragent-0.16.10
  warbler-2.0.4

--- Bundler
Bundler.bundle_path: /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems
Bundler.root: /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF
Gemfile: /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/Gemfile
Settings:
  gemfile = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/Gemfile
  without = [:development, :test, :assets]
  bin_path = /Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/bundler-1.16.1/exe/bundle

--- JRuby-Rack Config
compat_version = RUBY1_9
default_logger = org.jruby.rack.logging.StandardOutLogger@15676dfa
equals = <error: >
err = org.apache.tomcat.util.log.SystemLogHandler@a019ea7
filter_adds_html = true
filter_verifies_resource = false
ignore_environment = false
initial_memory_buffer_size = 
initial_runtimes = 1
jms_connection_factory = 
jms_jndi_properties = 
logger = org.jruby.rack.logging.Slf4jLogger@33108b1
logger_class_name = slf4j
logger_name = jruby.rack
maximum_memory_buffer_size = 
maximum_runtimes = 1
num_initializer_threads = 
out = org.apache.tomcat.util.log.SystemLogHandler@5c935cad
rackup = 
rackup_path = 
rewindable = true
runtime_arguments = 
runtime_environment = 
runtime_timeout_seconds = 
serial_initialization = false
servlet_context = org.apache.catalina.core.ApplicationContextFacade@4a8373ab
throw_init_exception = false

--- System Properties
awt.toolkit = sun.lwawt.macosx.LWCToolkit
catalina.base = /Users/teamcity/Applications/apache-tomcat
catalina.home = /Users/teamcity/Applications/apache-tomcat
catalina.useNaming = true
common.loader = ${catalina.base}/lib,${catalina.base}/lib/*.jar,${catalina.home}/lib,${catalina.home}/lib/*.jar
file.encoding = UTF-8
file.encoding.pkg = sun.io
file.separator = /
ftp.nonProxyHosts = local|*.local|169.254/16|*.169.254/16
gopherProxySet = false
http.nonProxyHosts = local|*.local|169.254/16|*.169.254/16
java.awt.graphicsenv = sun.awt.CGraphicsEnvironment
java.awt.printerjob = sun.lwawt.macosx.CPrinterJob
java.class.path = /Users/teamcity/Applications/apache-tomcat/bin/bootstrap.jar:/Users/teamcity/Applications/apache-tomcat/bin/tomcat-juli.jar
java.class.version = 52.0
java.endorsed.dirs = /Users/teamcity/Applications/apache-tomcat/endorsed
java.ext.dirs = /Users/teamcity/Library/Java/Extensions:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/ext:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java
java.home = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
java.io.tmpdir = /Users/teamcity/Applications/apache-tomcat/temp
java.library.path = /Users/teamcity/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
java.naming.factory.initial = org.apache.naming.java.javaURLContextFactory
java.naming.factory.url.pkgs = org.apache.naming
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.8.0_162-b12
java.specification.name = Java Platform API Specification
java.specification.vendor = Oracle Corporation
java.specification.version = 1.8
java.util.logging.config.file = /Users/teamcity/Applications/apache-tomcat/conf/logging.properties
java.util.logging.manager = org.apache.juli.ClassLoaderLogManager
java.vendor = Oracle Corporation
java.vendor.url = http://java.oracle.com/
java.vendor.url.bug = http://bugreport.sun.com/bugreport/
java.version = 1.8.0_162
java.vm.info = mixed mode
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.specification.name = Java Virtual Machine Specification
java.vm.specification.vendor = Oracle Corporation
java.vm.specification.version = 1.8
java.vm.vendor = Oracle Corporation
java.vm.version = 25.162-b12
jdk.tls.ephemeralDHKeySize = matched
jruby.backtrace.style = raw
jruby.cli.debug = true
jruby.debug.fullTrace = true
jruby.management.enabled = true
line.separator = 
logback.configurationFile = lib/properties/tomcat/logback.groovy
org.apache.catalina.startup.ContextConfig.jarsToSkip = 
org.apache.catalina.startup.TldConfig.jarsToSkip = tomcat7-websocket.jar
os.arch = x86_64
os.name = Mac OS X
os.version = 10.12.6
package.access = sun.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.naming.resources.,org.apache.tomcat.
package.definition = sun.,java.,org.apache.catalina.,org.apache.coyote.,org.apache.jasper.,org.apache.naming.,org.apache.tomcat.
path.separator = :
server.loader = 
shared.loader = 
socksNonProxyHosts = local|*.local|169.254/16|*.169.254/16
sun.arch.data.model = 64
sun.boot.class.path = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/resources.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/rt.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/sunrsasign.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/jsse.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/jce.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/charsets.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib/jfr.jar:/Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/classes
sun.boot.library.path = /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre/lib
sun.cpu.endian = little
sun.cpu.isalist = 
sun.io.unicode.encoding = UnicodeBig
sun.java.command = org.apache.catalina.startup.Bootstrap start
sun.java.launcher = SUN_STANDARD
sun.jnu.encoding = UTF-8
sun.management.compiler = HotSpot 64-Bit Tiered Compilers
sun.os.patch.level = unknown
tomcat.util.buf.StringCache.byte.enabled = true
tomcat.util.scan.DefaultJarScanner.jarsToSkip = bootstrap.jar,commons-daemon.jar,tomcat-juli.jar,annotations-api.jar,el-api.jar,jsp-api.jar,servlet-api.jar,websocket-api.jar,catalina.jar,catalina-ant.jar,catalina-ha.jar,catalina-tribes.jar,jasper.jar,jasper-el.jar,ecj-*.jar,tomcat-api.jar,tomcat-util.jar,tomcat-coyote.jar,tomcat-dbcp.jar,tomcat-jni.jar,tomcat-spdy.jar,tomcat-i18n-en.jar,tomcat-i18n-es.jar,tomcat-i18n-fr.jar,tomcat-i18n-ja.jar,tomcat-juli-adapters.jar,catalina-jmx-remote.jar,catalina-ws.jar,tomcat-jdbc.jar,tools.jar,commons-beanutils*.jar,commons-codec*.jar,commons-collections*.jar,commons-dbcp*.jar,commons-digester*.jar,commons-fileupload*.jar,commons-httpclient*.jar,commons-io*.jar,commons-lang*.jar,commons-logging*.jar,commons-math*.jar,commons-pool*.jar,jstl.jar,taglibs-standard-spec-*.jar,geronimo-spec-jaxrpc*.jar,wsdl4j*.jar,ant.jar,ant-junit*.jar,aspectj*.jar,jmx.jar,h2*.jar,hibernate*.jar,httpclient*.jar,jmx-tools.jar,jta*.jar,log4j.jar,log4j-1*.jar,mail*.jar,slf4j*.jar,xercesImpl.jar,xmlParserAPIs.jar,xml-apis.jar,junit.jar,junit-*.jar,hamcrest*.jar,org.hamcrest*.jar,ant-launcher.jar,cobertura-*.jar,asm-*.jar,dom4j-*.jar,icu4j-*.jar,jaxen-*.jar,jdom-*.jar,jetty-*.jar,oro-*.jar,servlet-api-*.jar,tagsoup-*.jar,xmlParserAPIs-*.jar,xom-*.jar
user.country = US
user.dir = /Users/teamcity/Applications/buildAgent/work/51de57e2559977e7
user.home = /Users/teamcity
user.language = en
user.name = teamcity
user.timezone = America/New_York

--- $LOAD_PATH:
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/lib
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/app/models/offer/
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/app/assets
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/app/controllers
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/app/mailers
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/app/models
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/gems/gems/jquery-rails-4.3.1/lib
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/gems/gems/jquery-rails-4.3.1/vendor
/Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT/WEB-INF/gems/gems/actionview-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/manifest-1.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/simplecov-html-0.10.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/simplecov-0.16.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/bundler-1.16.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/bundler/gems/warbler-848029258bd4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/useragent-0.16.10/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/state_machines-activerecord-0.5.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/state_machines-activemodel-0.5.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/state_machines-0.5.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/simplecov-teamcity-summary-0.1.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/bundler/gems/rest-client-components-236ac31ed1a0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rest-client-2.0.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rails-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/sprockets-rails-3.2.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/sprockets-3.7.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rack-cache-1.7.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/omniauth-google-oauth2-0.5.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/omniauth-oauth2-1.5.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/omniauth-1.8.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/oauth2-1.4.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/newrelic_rpm-4.8.0.341/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/netrc-0.11.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/json-2.1.0-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jruby-rack-1.1.21/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jruby-jars-9.1.16.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jquery-rails-4.3.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/method_source-0.9.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/bundler/gems/JGLib-aacbf72aec6c/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/thor-0.20.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/minitest-proveit-1.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/mimemagic-0.3.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jbundler-0.9.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/ruby-maven-3.3.12/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/ruby-maven-libs-3.3.9/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/maven-tools-1.1.6/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/virtus-1.0.5/lib
uri:classloader:/META-INF/jruby.home/lib/ruby/gems/shared/gems/jar-dependencies-0.3.12/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/httparty-0.16.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/multi_xml-0.6.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/http-cookie-1.0.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/hashie-3.5.7/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-cloud-bigquery-1.1.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-cloud-core-1.2.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-cloud-env-1.0.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-api-client-0.19.8/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-api-client-0.19.8/generated
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/google-api-client-0.19.8/third_party
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/retriable-3.1.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/representable-3.0.4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/uber-0.1.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/mime-types-3.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/mime-types-data-3.2016.0521/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/httpclient-2.8.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/googleauth-0.6.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/signet-0.8.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/os-0.9.6/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/memoist-0.16.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/logging-2.2.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/multi_json-1.13.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/little-plugger-1.1.4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jwt-1.5.6/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/geoip-1.6.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/faraday-0.12.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/multipart-post-2.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/equalizer-0.0.11/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/domain_name-0.5.20170404/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/unf-0.1.4-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/docile-1.3.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/declarative-option-0.1.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/declarative-0.0.10/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/bundler/gems/commander-befcd202c87f/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/coercible-1.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/axlsx-2.0.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rubyzip-1.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/htmlentities-4.3.4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/axiom-types-0.1.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/ice_nine-0.11.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/descendants_tracker-0.0.4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/aws-sdk-2.11.16/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/aws-sdk-resources-2.11.16/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/aws-sdk-core-2.11.16/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jmespath-1.3.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/aws-sigv4-1.0.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/ar-octopus-0.9.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/addressable-2.5.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/public_suffix-3.0.2/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activerecord-jdbcmysql-adapter-51.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/jdbc-mysql-5.1.44/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activerecord-jdbc-adapter-51.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activerecord-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/arel-8.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activemodel-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/actionmailer-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/mail-2.7.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/mini_mime-1.0.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activejob-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/globalid-0.4.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/actioncable-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/websocket-driver-0.6.5-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/websocket-extensions-0.1.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/nio4r-2.3.0-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/actionpack-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rack-test-0.8.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rack-2.0.4/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/actionview-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rails-html-sanitizer-1.0.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/loofah-2.2.0/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/crass-1.0.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rails-dom-testing-2.0.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/nokogiri-1.8.2-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/erubi-1.7.1/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/builder-3.2.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/tzinfo-1.2.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/thread_safe-0.3.6-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/minitest-5.10.3/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/i18n-0.9.5/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/concurrent-ruby-1.0.5-java/lib
/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/rake-12.3.0/lib
uri:classloader:/META-INF/jruby.home/lib/ruby/2.3/site_ruby
uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib
@headius

This comment has been minimized.

Member

headius commented Apr 10, 2018

Are you able to trigger this consistently? If possible, you can set JRuby into a debug mode that will show the actual cause of the LoadError better by setting property jruby.cli.debug=true (and I'll try to make LoadError do a better job of reporting.

Very little of JRuby internals would trigger this exception, since most of the time we're using Ruby collection types that raise Ruby errors rather than Java errors like ConcurrentModificationException. With debug mode on we should be able to see what's happening.

It would also be helpful if you can make a sample project that we can build and run to reproduce this. A fix can be found within hours if not minutes when we can reproduce the issue locally.

@headius headius added this to the JRuby 9.1.17.0 milestone Apr 10, 2018

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 10, 2018

Did I do it wrong? I see jruby.cli.debug=true set to true already in the backtrace above?

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

Oh yes, you did! I only looked at the servlet context...did not see the full system properties later.

Unfortunately it appears that the error went to a different log. I believe we always dump these Java exceptions during load to whatever the error stream of the JRuby runtime is. Offhand I do not recall where that goes with jruby-rack. Assuming it did print, you should be able to grep other logs for ConcurrentModificationException and hopefully find the original trace. Otherwise, we'll have to figure out why it's not logging anywhere.

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 12, 2018

Found the following in catalina.out:

Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server version:        Apache Tomcat/7.0.62
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server built:          May 7 2015 17:14:55 UTC
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Server number:         7.0.62.0
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Name:               Mac OS X
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: OS Version:            10.12.6
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Architecture:          x86_64
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Java Home:             /Library/Java/JavaVirtualMachines/jdk1.8.0_162.jdk/Contents/Home/jre
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Version:           1.8.0_162-b12
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: JVM Vendor:            Oracle Corporation
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_BASE:         /Users/teamcity/Applications/apache-tomcat
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: CATALINA_HOME:         /Users/teamcity/Applications/apache-tomcat
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.util.logging.config.file=/Users/teamcity/Applications/apache-tomcat/conf/logging.properties
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Xmx2G
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djruby.debug.fullTrace=true
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djruby.cli.debug=true
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djruby.backtrace.style=raw
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.endorsed.dirs=/Users/teamcity/Applications/apache-tomcat/endorsed
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.base=/Users/teamcity/Applications/apache-tomcat
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Dcatalina.home=/Users/teamcity/Applications/apache-tomcat
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.VersionLoggerListener log
INFO: Command line argument: -Djava.io.tmpdir=/Users/teamcity/Applications/apache-tomcat/temp
Mar 27, 2018 12:44:52 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /Users/teamcity/Library/Java/Extensions:/Library/Java/Extensions:/Network/Library/Java/Extensions:/System/Library/Java/Extensions:/usr/lib/java:.
Mar 27, 2018 12:44:52 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-8080"]
Mar 27, 2018 12:44:52 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-bio-8009"]
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 844 ms
Mar 27, 2018 12:44:52 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Mar 27, 2018 12:44:52 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.62
Mar 27, 2018 12:44:52 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deploying web application archive /Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT.war
Mar 27, 2018 12:45:06 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Using Ext extension for JSON.
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- rest-client-components
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- idn
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- idn
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- rubyzip
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- nokogiri/2.3/nokogiri
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- io/extra
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- google-api-client
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- dotenv-rails
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- puma
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- linecache
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- factory_bot_rails
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- concurrent-ruby
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- net-http-digest_auth
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- ruby-hmac
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- selenium-webdriver
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- minitest-rails
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- minitest-reporters
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- minitest-retry
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- mocha
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- rails-perftest
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- rails-controller-testing
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- tzinfo/data
Exception `NameError' at java/lang/Thread.java:1559 - uninitialized constant Offer::Campaign
Exception `NameError' at java/lang/Thread.java:1559 - uninitialized constant OfferGroup::Campaign
Exception `NameError' at java/lang/Thread.java:1559 - uninitialized constant WebTrafficUrl::Offer
Exception `NameError' at java/lang/Thread.java:1559 - uninitialized constant OfferCounter::Offer
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- oj
Exception `Aws::S3::Errors::NoSuchKey' at java/lang/Thread.java:1559 - The specified key does not exist.
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- ox
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- ox
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- oga
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- oga
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- libxml
Exception `LoadError' at java/lang/Thread.java:1559 - no such file to load -- libxml
java.util.ConcurrentModificationException
	at java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:806)
	at java.util.WeakHashMap$KeyIterator.next(WeakHashMap.java:839)
	at org.jruby.RubyClass.invalidateCacheDescendants(RubyClass.java:1201)
	at org.jruby.RubyModule.includeModule(RubyModule.java:889)
	at org.jruby.RubyModule.append_features(RubyModule.java:2596)
	at org.jruby.RubyModule$INVOKER$i$1$0$append_features_DBG.call(RubyModule$INVOKER$i$1$0$append_features_DBG.gen)
	at org.jruby.RubyClass.finvoke(RubyClass.java:908)
	at org.jruby.runtime.Helpers.invoke(Helpers.java:442)
	at org.jruby.RubyBasicObject.callMethod(RubyBasicObject.java:363)
	at org.jruby.RubyModule.include(RubyModule.java:2639)
	at org.jruby.RubyModule$INVOKER$i$include_DBG.call(RubyModule$INVOKER$i$include_DBG.gen)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.InterpretedIRBodyMethod.interpretWithBacktrace(InterpretedIRBodyMethod.java:72)
	at org.jruby.internal.runtime.methods.InterpretedIRBodyMethod.INTERPRET_CLASS(InterpretedIRBodyMethod.java:66)
	at org.jruby.internal.runtime.methods.InterpretedIRBodyMethod.callInternal(InterpretedIRBodyMethod.java:48)
	at org.jruby.internal.runtime.methods.InterpretedIRBodyMethod.call(InterpretedIRBodyMethod.java:37)
	at org.jruby.ir.instructions.ProcessModuleBodyInstr.interpret(ProcessModuleBodyInstr.java:56)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.processOtherOp(StartupInterpreterEngine.java:178)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:104)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:107)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:94)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:34)
	at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
	at org.jruby.Ruby.runInterpreter(Ruby.java:842)
	at org.jruby.Ruby.loadFile(Ruby.java:2891)
	at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
	at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
	at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:891)
	at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:531)
	at org.jruby.runtime.load.LoadService.require(LoadService.java:402)
	at org.jruby.RubyKernel.requireCommon(RubyKernel.java:963)
	at org.jruby.RubyKernel.require19(RubyKernel.java:956)
	at org.jruby.RubyKernel$INVOKER$s$1$0$require19_DBG.call(RubyKernel$INVOKER$s$1$0$require19_DBG.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneBlock.call(JavaMethod.java:637)
	at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:101)
	at org.jruby.ir.runtime.IRRuntimeHelpers.unresolvedSuper(IRRuntimeHelpers.java:1042)
	at org.jruby.ir.runtime.IRRuntimeHelpers.zSuper(IRRuntimeHelpers.java:1055)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.activesupport_minus_5_dot_1_dot_5.lib.active_support.dependencies.invokeSuper2:-unknown-super-target-(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.activesupport_minus_5_dot_1_dot_5.lib.active_support.dependencies.RUBY$block$require$1(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
	at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:80)
	at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yieldSpecific(IRRuntimeHelpers.java:421)
	at org.jruby.ir.targets.YieldSite.yieldSpecific(YieldSite.java:114)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.activesupport_minus_5_dot_1_dot_5.lib.active_support.dependencies.RUBY$method$load_dependency$0(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:258)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:163)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:170)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.activesupport_minus_5_dot_1_dot_5.lib.active_support.dependencies.invokeOther4:load_dependency(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.activesupport_minus_5_dot_1_dot_5.lib.active_support.dependencies.RUBY$method$require$0(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/activesupport-5.1.5/lib/active_support/dependencies.rb:292)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:103)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:163)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:202)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:107)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:94)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:34)
	at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
	at org.jruby.Ruby.runInterpreter(Ruby.java:842)
	at org.jruby.Ruby.loadFile(Ruby.java:2891)
	at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
	at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
	at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:891)
	at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:531)
	at org.jruby.runtime.load.LoadService.autoloadRequire(LoadService.java:406)
	at org.jruby.RubyKernel$1.load(RubyKernel.java:196)
	at org.jruby.RubyModule$Autoload.getConstant(RubyModule.java:4625)
	at org.jruby.RubyModule.getAutoloadConstant(RubyModule.java:4422)
	at org.jruby.RubyModule.getAutoloadConstant(RubyModule.java:4415)
	at org.jruby.RubyModule.resolveUndefConstant(RubyModule.java:3828)
	at org.jruby.RubyModule.getConstantFromNoConstMissing(RubyModule.java:3797)
	at org.jruby.ir.instructions.SearchModuleForConstInstr.cache(SearchModuleForConstInstr.java:73)
	at org.jruby.ir.instructions.SearchModuleForConstInstr.interpret(SearchModuleForConstInstr.java:103)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.processOtherOp(StartupInterpreterEngine.java:178)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:104)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:127)
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:149)
	at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:186)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:116)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.RubyKernel.tap(RubyKernel.java:1765)
	at org.jruby.RubyKernel$INVOKER$s$0$0$tap_DBG.call(RubyKernel$INVOKER$s$0$0$tap_DBG.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:536)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:429)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:127)
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:149)
	at org.jruby.runtime.IRBlockBody.yieldSpecific(IRBlockBody.java:84)
	at org.jruby.runtime.Block.yieldSpecific(Block.java:134)
	at org.jruby.ext.thread.Mutex.synchronize(Mutex.java:158)
	at org.jruby.ext.thread.Mutex$INVOKER$i$0$0$synchronize_DBG.call(Mutex$INVOKER$i$0$0$synchronize_DBG.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:536)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:429)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:56)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:127)
	at org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:149)
	at org.jruby.runtime.IRBlockBody.doYield(IRBlockBody.java:193)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:125)
	at org.jruby.runtime.Block.yieldNonArray(Block.java:169)
	at org.jruby.RubyBasicObject.yieldUnder(RubyBasicObject.java:1751)
	at org.jruby.RubyBasicObject.instance_exec19(RubyBasicObject.java:1728)
	at org.jruby.RubyBasicObject$INVOKER$i$0$3$instance_exec19_DBG.call(RubyBasicObject$INVOKER$i$0$3$instance_exec19_DBG.gen)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:77)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.railties_minus_5_dot_1_dot_5.lib.rails.initializable.invokeOther2:instance_exec(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:30)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.railties_minus_5_dot_1_dot_5.lib.rails.initializable.RUBY$method$run$0(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:30)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:176)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:67)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.railties_minus_5_dot_1_dot_5.lib.rails.initializable.invokeOther1:run(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:59)
	at Users.teamcity.Applications.apache_minus_tomcat.webapps.ROOT.WEB_minus_INF.gems.gems.railties_minus_5_dot_1_dot_5.lib.rails.initializable.RUBY$block$run_initializers$0(/Users/teamcity/Applications/apache-tomcat/webapps/ROOT/WEB-INF/gems/gems/railties-5.1.5/lib/rails/initializable.rb:59)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
	at org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:123)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:417)
	at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:87)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.tsort.RUBY$block$tsort_each$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:228)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
	at org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:123)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:417)
	at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:87)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.tsort.RUBY$block$each_strongly_connected_component$1(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:350)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.ir.runtime.IRRuntimeHelpers.yield(IRRuntimeHelpers.java:417)
	at org.jruby.ir.targets.YieldSite.yield(YieldSite.java:87)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.tsort.RUBY$method$each_strongly_connected_component_from$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:431)
	at org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:77)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:93)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:77)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.tsort.invokeOther2:each_strongly_connected_component_from(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:349)
	at uri_3a_classloader_3a_.META_minus_INF.jruby_dot_home.lib.ruby.stdlib.tsort.RUBY$block$each_strongly_connected_component$0(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/tsort.rb:349)
	at org.jruby.runtime.CompiledIRBlockBody.yieldDirect(CompiledIRBlockBody.java:156)
	at org.jruby.runtime.MixedModeIRBlockBody.yieldDirect(MixedModeIRBlockBody.java:123)
	at org.jruby.runtime.BlockBody.yield(BlockBody.java:114)
	at org.jruby.runtime.Block.yield(Block.java:165)
	at org.jruby.RubyArray.each(RubyArray.java:1735)
	at org.jruby.RubyArray$INVOKER$i$0$0$each_DBG.call(RubyArray$INVOKER$i$0$0$each_DBG.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodZeroBlock.call(JavaMethod.java:536)
	at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:101)
	at org.jruby.RubyMethod.call(RubyMethod.java:127)
	at org.jruby.RubyMethod$INVOKER$i$call_DBG.call(RubyMethod$INVOKER$i$call_DBG.gen)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:429)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:429)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:432)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:109)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:95)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:278)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:79)
	at org.jruby.runtime.callsite.CachingCallSite.callIter(CachingCallSite.java:84)
	at org.jruby.ir.instructions.CallBase.interpret(CallBase.java:429)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:360)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:90)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:214)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:200)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:210)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:338)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:183)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:324)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:107)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:94)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:34)
	at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
	at org.jruby.Ruby.runInterpreter(Ruby.java:842)
	at org.jruby.Ruby.loadFile(Ruby.java:2891)
	at org.jruby.runtime.load.LibrarySearcher$ResourceLibrary.load(LibrarySearcher.java:243)
	at org.jruby.runtime.load.LibrarySearcher$FoundLibrary.load(LibrarySearcher.java:34)
	at org.jruby.runtime.load.LoadService.tryLoadingLibraryOrScript(LoadService.java:891)
	at org.jruby.runtime.load.LoadService.smartLoadInternal(LoadService.java:531)
	at org.jruby.runtime.load.LoadService.require(LoadService.java:402)
	at org.jruby.RubyKernel.requireCommon(RubyKernel.java:963)
	at org.jruby.RubyKernel.require19(RubyKernel.java:956)
	at org.jruby.RubyKernel$INVOKER$s$1$0$require19_DBG.call(RubyKernel$INVOKER$s$1$0$require19_DBG.gen)
	at org.jruby.internal.runtime.methods.JavaMethod$JavaMethodOneOrNBlock.call(JavaMethod.java:398)
	at org.jruby.internal.runtime.methods.AliasMethod.call(AliasMethod.java:61)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:318)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:155)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:315)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:78)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:144)
	at org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:130)
	at org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:194)
	at org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:298)
	at org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:127)
	at org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:344)
	at org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)
	at org.jruby.ir.interpreter.Interpreter.INTERPRET_ROOT(Interpreter.java:107)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:94)
	at org.jruby.ir.interpreter.Interpreter.execute(Interpreter.java:34)
	at org.jruby.ir.IRTranslator.execute(IRTranslator.java:42)
	at org.jruby.Ruby.evalScriptlet(Ruby.java:482)
	at org.jruby.Ruby.evalScriptlet(Ruby.java:458)
	at org.jruby.rack.rails.RailsRackApplicationFactory.createApplicationObject(RailsRackApplicationFactory.java:22)
	at org.jruby.rack.DefaultRackApplicationFactory$1.create(DefaultRackApplicationFactory.java:101)
	at org.jruby.rack.DefaultRackApplicationFactory$RackApplicationImpl.init(DefaultRackApplicationFactory.java:445)
	at org.jruby.rack.DefaultRackApplicationFactory.getApplication(DefaultRackApplicationFactory.java:114)
	at org.jruby.rack.SharedRackApplicationFactory.doInit(SharedRackApplicationFactory.java:34)
	at org.jruby.rack.RackApplicationFactoryDecorator.init(RackApplicationFactoryDecorator.java:100)
	at org.jruby.rack.RackServletContextListener.contextInitialized(RackServletContextListener.java:50)
	at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5016)
	at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5528)
	at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)
	at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901)
	at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877)
	at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
	at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1095)
	at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1930)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Exception `LoadError' at java/lang/Thread.java:1559 - load error: pp -- java.util.ConcurrentModificationException: null
Mar 27, 2018 12:45:49 PM org.apache.catalina.startup.HostConfig deployWAR
INFO: Deployment of web application archive /Users/teamcity/Applications/apache-tomcat-7.0.62/webapps/ROOT.war has finished in 57,201 ms
Mar 27, 2018 12:45:49 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-8080"]
Mar 27, 2018 12:45:49 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-bio-8009"]
Mar 27, 2018 12:45:49 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 57294 ms
@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

I don't believe I asked...is this a recent regression? In other words, did it work in 9.1.15 or an earlier version?

The exception you found is exactly what I was looking for. Something about the way we are invalidating a particular class hierarchy is causing a concurrency error, either because it is not sufficiently protecting itself from other threads, or because it is improperly manipulating some collection (like deleting while iterating, or similar).

So it seems under some circumstance, requiring the pp library is causing a problem.

I am wondering about the threading characteristics of jruby-rack these days. It is possible that a request is being handled by a runtime at the same time it is loading the pp library, and that's triggering some concurrent modification of the same class hierarchy. This might happen if there are autoloaded libraries in play. You may be able to see what library is triggering the load of pp by setting jruby.debug.loadService.

Obviously the pp library does not fail for me locally, nor have I seen this reported before, even under very similar concurrent-loading situations. So you have found a good one to fix 👍

Is it possible for you to create a dummy application that mimics yours and fails the same way? It would help a great deal if I could reproduce this locally.

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 12, 2018

The master branch of our application is on 9.1.7.0 and I haven't seen it there, however I'm in the process of upgrading said app from Rails 4.2 to 5.1, and upgrading JRuby came along for that ride, so there have been many shallow changes from the known stable configuration.

Does jruby.compat.version = 1.9 do anything (good?) in JRuby 9K? I am pretty sure that has been in our warbler.rb for years (dating back to when we needed to specify it to avoid getting 1.8) but looking at it now (and noticing that require19 appears in the stack trace) I wonder if it's a contributing factor....

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

No, jruby.compat.version does not do anything anymore (ignored), and JRuby 9.1 supports only Ruby 2.3.

Are you able to boot the Rails app on its own, outside of Tomcat?

At this point the best I can do is to eyeball the related code and see if I can divine a path through it that might lead to this exception. The big missing bits of information are:

  • how many threads might be attempting to load library code concurrently
  • what library is triggering the load of pp

Perhaps you can try to configure your jruby-rack to only have use a separate instance for each request, and see if that solves it? I don't recommend that pattern for deploying, since we and Rails should be safe for concurrent requests, but it would help narrow down the problem.

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

I may see how this error could be triggered. The concurrent modification is happening against a set of subclasses. That set is itself synchronized (on a per-call basis) but iteration would not be covered by that lock. A thread iterating this list while a subclass is simultaneously removed could trigger the error.

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 12, 2018

I just discovered Rails.application.executor.wrap last night (how did it not get so much as mention in Rails' 5.0 upgrade guide?!?), so at the moment I'm pursuing this as a possible manifestation of a race between threads at startup autoloading things in an unsafe way. I've updated the same branch where I've already upgraded JRuby and Rails to wrap all of our explicitly created threads' bodies with this executor. I'll report back on whether or not that makes any impact on the frequency that this pops up.

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

@JasonLunn Ok, that sounds like a good lead.

I have also managed to easily reproduce this same exception by following my hunch above:

class A; def foo; end; end
Thread.new { loop { Class.new(A) } }
loop { class A; def foo; end; end }

This script almost immediately produces the same exception you reported:

Unhandled Java exception: java.util.ConcurrentModificationException
java.util.ConcurrentModificationException: null
                   nextEntry at java/util/WeakHashMap.java:806
                        next at java/util/WeakHashMap.java:839
  invalidateCacheDescendants at org/jruby/RubyClass.java:1203
...

I am exploring possible fixes. This is a touchy area, so I can't just go throwing locks around.

headius added a commit that referenced this issue Apr 12, 2018

Use ConcurrentWeakHashMap for RubyClass.subclasses.
The naively-synchronized WeakHashSet does not guarantee safe
iteration in the presence of modification, which triggers a
ConcurrentModificationException as seen in #5115.

By using a ConcurrentWeakHashMap (designed by Doug Lea et al, see
http://www.java2s.com/Code/Java/Collections-Data-Structure/Ahashtablewithemweakkeysemfullconcurrencyofretrievalsandadjustableexpectedconcurrencyforupdates.htm)
we can iterate and modify concurrently without errors.

However this implementation is based the design of
ConcurrentHashMap, which means the in-memory structure will likely
be considerably larger than a simple WeakHashMap. I believe it is
also based off the pre-Java 8 ConcurrentHashMap, which means any
load factor above one thread greatly increases in-memory size, and
a load factor of one limits the concurrent use of the collection.
@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

I've made a PR using a public domain "ConcurrentWeakHashMap" I found that was designed by the java.util.concurrent folks. It allows concurrent traversal and modification, but there are some caveats.

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

Oh, I also tried another approach on JRuby master: using Java 8 streams to iterate through the subclasses set rather than external iteration. It did not appear to fix the bug.

@headius headius closed this Apr 12, 2018

@headius

This comment has been minimized.

Member

headius commented Apr 12, 2018

Ok, you should be able to see this in 9.1 nightlies, though it may be tricky to use them with Warbler until we release 9.1.17. I'm pretty confident this will fix your issue.

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 12, 2018

When is the next JRubyConf? I owe you two beers this release cycle...

@JasonLunn

This comment has been minimized.

JasonLunn commented Apr 18, 2018

Is Cloudbees' Jenkins down for anyone else? I wanted to try the nightly to see if it would work for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment