Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Internal Server Error in Environment API Calls Version 19.1.0 #5807

Closed
justinmathewbiji opened this issue Feb 6, 2019 · 5 comments

Comments

Projects
6 participants
@justinmathewbiji
Copy link

commented Feb 6, 2019

Bug Report
Summary

We are creating the environments from config-repo, everything was working fine in the old version, 18.12.0. But after upgrading gocd to 19.1.0, we are facing the problems in all Environment related API calls.

Environment

GoCD environment using is 19.1.0. We are getting internal server error occurred in the environment related api calls if and only if we have atleast one environment created from code (environment as code, in config-repo)
API calls like
/go/api/admin/environments

Basic environment details
  • Go Version: 19.1.0
  • JAVA Version: 1.8.0_181
  • OS: Linux 3.10.0-862.11.6.el7.x86_64

Steps to regenerate the issue:

  1. Create an environment using environment as code (yaml plugin) [https://github.com/tomzo/gocd-yaml-config-plugin]
  2. make sure that the environment is visible in UI
  3. Try to call API call either from Browser, Code or postman
Expected Results

service must return the environment details response

Actual Results

500 Internal Server Error

Any other info

This issue will happen only when you have atleast one environment generated from config-repo.

Log snippets

origin=com.thoughtworks.go.config.remote.FileConfigOrigin@5b9b
]],agents=[com.thoughtworks.go.config.EnvironmentAgentConfig@cf87262c, com.thoughtworks.go.config.EnvironmentAgentConfig@f08a1032],
pipelines=[],configErrors={},origin=com.thoughtworks.go.config.remote.FileConfigOrigin@5b9b],
com.thoughtworks.go.config.BasicEnvironmentConfig@39d4a068[name=CD_Admin,variables=[],agents=[],
pipelines=[com.thoughtworks.go.config.EnvironmentPipelineConfig@e3a883ae],configErrors={},
origin=git@*****:CDP/MasterConfigRepo.git at0a9e6292d4839fbbdb7b788b76f2b2dd5f1b6f8b]] does not have a ConfigTag

Detailed Log

origin=com.thoughtworks.go.config.remote.FileConfigOrigin@5b9b
]],agents=[com.thoughtworks.go.config.EnvironmentAgentConfig@cf87262c, com.thoughtworks.go.config.EnvironmentAgentConfig@f08a1032],pipelines=[],configErrors={},origin=com.thoughtworks.go.config.remote.FileConfigOrigin@5b9b], com.thoughtworks.go.config.BasicEnvironmentConfig@39d4a068[name=CD_Admin,variables=[],agents=[],pipelines=[com.thoughtworks.go.config.EnvironmentPipelineConfig@e3a883ae],configErrors={},origin=git@git.allianz.co.id:CDP/MasterConfigRepo.git at0a9e6292d4839fbbdb7b788b76f2b2dd5f1b6f8b]] does not have a ConfigTag
at com.thoughtworks.go.util.ExceptionUtils.bomb(ExceptionUtils.java:30)
at com.thoughtworks.go.util.ExceptionUtils.bombIf(ExceptionUtils.java:58)
at com.thoughtworks.go.config.MagicalGoConfigXmlWriter.toXmlPartial(MagicalGoConfigXmlWriter.java:109)
at com.thoughtworks.go.server.service.EntityHashingService.computeMd5For(EntityHashingService.java:203)
at com.thoughtworks.go.server.service.EntityHashingService.getFromCache(EntityHashingService.java:180)
at com.thoughtworks.go.server.service.EntityHashingService.md5ForEntity(EntityHashingService.java:95)
at com.thoughtworks.go.apiv2.environments.EnvironmentsControllerV2.etagFor(EnvironmentsControllerV2.java:182)
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1382)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)
at com.thoughtworks.go.apiv2.environments.EnvironmentsControllerV2.calculateEtag(EnvironmentsControllerV2.java:219)
at com.thoughtworks.go.apiv2.environments.EnvironmentsControllerV2.index(EnvironmentsControllerV2.java:97)
at spark.RouteImpl$1.handle(RouteImpl.java:72)
at spark.http.matching.Routes.execute(Routes.java:61)
at spark.http.matching.MatcherFilter.doFilter(MatcherFilter.java:130)
at spark.servlet.SparkFilter.doFilter(SparkFilter.java:173)
at com.thoughtworks.go.spark.SparkPreFilter.doFilter(SparkPreFilter.java:53)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:420)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:566)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:203)
at org.eclipse.jetty.server.Dispatcher.forward(Dispatcher.java:73)
at org.tuckey.web.filters.urlrewrite.NormalRewrittenUrl.doRewrite(NormalRewrittenUrl.java:213)
at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171)
at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145)
at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92)
at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:381)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.web.FlashLoadingFilter.doFilterInternal(FlashLoadingFilter.java:39)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.newsecurity.filters.DenyIfRefererIsNotFilesFilter.doFilterInternal(DenyIfRefererIsNotFilesFilter.java:53)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.newsecurity.filters.VerifyAuthorityFilter.doFilterInternal(VerifyAuthorityFilter.java:58)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.newsecurity.filters.AbstractUserEnabledCheckFilter.doFilterInternal(AbstractUserEnabledCheckFilter.java:67)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.ThreadLocalUserFilter.doFilterInternal(ThreadLocalUserFilter.java:42)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.newsecurity.filters.AbstractBasicAuthenticationFilter.filterWhenSecurityEnabled(AbstractBasicAuthenticationFilter.java:99)
at com.thoughtworks.go.server.newsecurity.filters.AbstractBasicAuthenticationFilter.doFilterInternal(AbstractBasicAuthenticationFilter.java:68)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.AssumeAnonymousUserFilter.doFilterInternal(AssumeAnonymousUserFilter.java:65)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.AbstractReAuthenticationFilter.doFilterInternal(AbstractReAuthenticationFilter.java:75)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.InvalidateAuthenticationOnSecurityConfigChangeFilter.doFilterInternal(InvalidateAuthenticationOnSecurityConfigChangeFilter.java:84)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:208)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
at com.thoughtworks.go.server.newsecurity.filters.AlwaysCreateSessionFilter.doFilterInternal(AlwaysCreateSessionFilter.java:41)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.AbstractSessionReduceIdleTimeoutFilter.doFilterInternal(AbstractSessionReduceIdleTimeoutFilter.java:45)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:185)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at com.thoughtworks.go.server.newsecurity.filters.ModeAwareFilter.doFilter(ModeAwareFilter.java:80)
at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:214)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:177)
at com.thoughtworks.go.server.newsecurity.filterchains.MainFilterChain.doFilter(MainFilterChain.java:76)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:347)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:263)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at com.thoughtworks.go.server.web.BackupFilter.doFilter(BackupFilter.java:81)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1637)
at com.thoughtworks.go.server.web.DefaultHeadersFilter.doFilter(DefaultHeadersFilter.java:50)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1629)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:527)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1253)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1155)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)
at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
at org.eclipse.jetty.server.Server.handle(Server.java:530)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:347)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:256)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:247)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce(EatWhatYouKill.java:140)
at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:382)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:708)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:626)
at java.lang.Thread.run(Thread.java:748)

@maheshp maheshp added this to the Release 19.2.0 milestone Feb 6, 2019

@maheshp

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

@justinmathewbiji the Environments API had some performance issues as reported in #5698. To fix this we had to re-implement the endpoint which seems to have introduced this issue while referring environments defined in ConfigRepo.

While we fix this issue, you should be able to switch back to older implementation by turning on the toggle using the below command.

curl 'http://<your-server-url>/go/api/admin/feature_toggles/use_old_environments_api' \
      -u 'username:password' \
      -H 'Confirm: true' \
      -H 'Content-Type: application/json' \
      -X POST -d '{"toggle_value": "on"}'

@maheshp maheshp added this to To do in 19.2.0 Feb 6, 2019

@bdpiparva bdpiparva assigned bdpiparva and unassigned bdpiparva Feb 6, 2019

@arvindsv arvindsv moved this from To do to In progress in 19.2.0 Feb 6, 2019

@justinmathewbiji

This comment has been minimized.

Copy link
Author

commented Feb 6, 2019

@maheshp Thank you so much for your response. Given workaround seems fine. Hopefully this bug will be resolved in the upcoming release.

@maheshp

This comment has been minimized.

Copy link
Member

commented Feb 6, 2019

Hopefully this bug will be resolved in the upcoming release.

@justinmathewbiji yes, this will be fixed as part of upcoming 19.2.0 release.

@bdpiparva

This comment has been minimized.

Copy link
Contributor

commented Feb 7, 2019

@justinmathewbiji - I have opened the PR #5812 to fix the issue.

@maheshp maheshp moved this from In progress to Done in 19.2.0 Feb 11, 2019

@rajiesh

This comment has been minimized.

Copy link
Contributor

commented Feb 14, 2019

Verified on 19.2.0 (8530-e1937046b7e1d9471b20ed24f6225cb24ef98ec0).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.