Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ResourceManager.cluster_application_kill() is not working #75

Closed
kevin-bates opened this issue Jan 11, 2020 · 7 comments · Fixed by #78
Closed

ResourceManager.cluster_application_kill() is not working #75

kevin-bates opened this issue Jan 11, 2020 · 7 comments · Fixed by #78

Comments

@kevin-bates
Copy link
Member

In troubleshooting issues in another repo, I found that ResourceManager.clsuter_application_kill() is resulting in 500 status code with the following message: "Response finished with status: 500. Details: "

However, using CURL or REST clients that issue calls, result in the expected results...
curl -X PUT -H 'Content-Type: application/json' -i 'http://yarn-eg-node-1.fyre.ibm.com:8088/ws/v1/cluster/apps/application_1574459664527_0116/state' --data '{"state":"KILLED"}'

All portions of the request appear to match what is used in curl, so I'm hoping someone has ideas about this.

I've also included a python file that submits an application, checks its state and attempts to kill via the resource manager. Upon determining the resource manager fails, it issues the kill call using curl, then confirms the state is KILLED.
((Please note that this python test uncovered an issue with cluster_submit_application() in which the response of no content could not be processed via response.json(). I will be submitting a PR for that issue shortly.))

Looking at the YARN logs, they contain the following traceback ...

2020-01-10 12:36:54,620 ERROR mortbay.log (Slf4jLog.java:warn(87)) - Error for /ws/v1/cluster/apps/application_1574459664527_0098/state
java.lang.Error: Error: could not match input
	at com.sun.jersey.json.impl.reader.JsonLexer.zzScanError(JsonLexer.java:491)
	at com.sun.jersey.json.impl.reader.JsonLexer.yylex(JsonLexer.java:736)
	at com.sun.jersey.json.impl.reader.JsonXmlStreamReader.nextToken(JsonXmlStreamReader.java:160)
	at com.sun.jersey.json.impl.reader.JsonXmlStreamReader.readNext(JsonXmlStreamReader.java:187)
	at com.sun.jersey.json.impl.reader.JsonXmlStreamReader.readNext(JsonXmlStreamReader.java:178)
	at com.sun.jersey.json.impl.reader.JsonXmlStreamReader.<init>(JsonXmlStreamReader.java:149)
	at com.sun.jersey.json.impl.Stax2JsonFactory.createReader(Stax2JsonFactory.java:134)
	at com.sun.jersey.json.impl.Stax2JsonFactory.createReader(Stax2JsonFactory.java:111)
	at com.sun.jersey.json.impl.BaseJSONUnmarshaller.createXmlStreamReader(BaseJSONUnmarshaller.java:113)
	at com.sun.jersey.json.impl.BaseJSONUnmarshaller.unmarshalJAXBElementFromJSON(BaseJSONUnmarshaller.java:108)
	at com.sun.jersey.json.impl.BaseJSONUnmarshaller.unmarshalFromJSON(BaseJSONUnmarshaller.java:97)
	at com.sun.jersey.json.impl.provider.entity.JSONRootElementProvider.readFrom(JSONRootElementProvider.java:125)
	at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.readFrom(AbstractRootElementProvider.java:111)
	at com.sun.jersey.spi.container.ContainerRequest.getEntity(ContainerRequest.java:474)
	at com.sun.jersey.server.impl.model.method.dispatch.EntityParamDispatchProvider$EntityInjectable.getValue(EntityParamDispatchProvider.java:123)
	at com.sun.jersey.server.impl.inject.InjectableValuesProvider.getInjectableValues(InjectableValuesProvider.java:46)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$EntityParamInInvoker.getParams(AbstractResourceMethodDispatchProvider.java:153)
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:203)
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:886)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:834)
	at org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebAppFilter.doFilter(RMWebAppFilter.java:178)
	at com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:795)
	at com.google.inject.servlet.FilterDefinition.doFilter(FilterDefinition.java:163)
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:58)
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.security.http.XFrameOptionsFilter.doFilter(XFrameOptionsFilter.java:57)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:109)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:617)
	at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:576)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.security.http.CrossOriginFilter.doFilter(CrossOriginFilter.java:95)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1400)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.apache.hadoop.http.NoCacheFilter.doFilter(NoCacheFilter.java:45)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:410)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
@kevin-bates kevin-bates mentioned this issue Jan 11, 2020
Closed
@kevin-bates
Copy link
Member Author

((Please note that this python test uncovered an issue with cluster_submit_application() in which the response of no content could not be processed via response.json(). I will be submitting a PR for that issue shortly.))

PR #77 should resolve the application submission issue if anyone wants to try the included python code to reproduce the kill issue.

@dimon222
Copy link
Collaborator

dimon222 commented Jan 11, 2020 •
edited
Loading

Brief look over stacktrace seem to refer to JAXB mapping of input JSON body. Perhaps headers for application/json are not passed for some reason? So body is not considered proper json.

@dimon222
Copy link
Collaborator

Can you also confirm your version of requests installed on test machine?

@kevin-bates
Copy link
Member Author

@dimon222 - thanks for the response.

Yeah, one of the google hits indicated a missing application/json entry in the header, but I confirmed that we're sending that into the requests method. Also, given the curl and REST clients also hit the same endpoint, indicates there must be something getting whacked in requests.

This issue reproduces using requests versions of 2.21.0 and 2.22.0. Haven't tried others.

@dimon222
Copy link
Collaborator

dimon222 commented Jan 11, 2020 •
edited
Loading

@kevin-bates try in these arguments to change data=data to json=data
https://github.com/toidi/hadoop-yarn-api-python-client/blob/master/yarn_api_client/resource_manager.py#L340

@dimon222
Copy link
Collaborator

dimon222 commented Jan 11, 2020 •
edited
Loading

I've tested it. It works using ^ what i mentioned there. I haven't checked the other methods, but its possible that other data= usages might be affected by same issue

@dimon222 dimon222 mentioned this issue Jan 11, 2020
Merged
@kevin-bates
Copy link
Member Author

Excellent! Thank you for the PR.

@dimon222 dimon222 mentioned this issue Jan 11, 2020
Open
@kevin-bates kevin-bates mentioned this issue Jan 14, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix json body request apis dimon222/hadoop-yarn-api-python-client
2 participants
@dimon222 @kevin-bates