HDDS-6419. Provide better error message for malformed auth header #3167
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@adoroszlai @elek Could you help to review this PR? |
adoroszlai
requested changes
Mar 5, 2022
There was a problem hiding this comment.
Thanks @symious for improving S3 Gateway.
- List bucket with empty auth.
- List bucket with OM offline
I think we should only fix the first case. We should not expose any internal details (e.g. OM address or state). Such problem should be inspected by admin.
|
@adoroszlai Thanks for the review. I agree we should keep some internal information away from end users. But we do have some cases to let users know the error themselves, I think we'll do some walkaround internally. Updated the patch, could you have a check? |
adoroszlai
reviewed
Mar 5, 2022
There was a problem hiding this comment.
Thanks @symious for updating the patch. Regarding client-side behavior, I think it is fine.
However, I think malformed authorization header is a user input problem, we should handle it as "expected" case. Logging verbose stack trace on S3 Gateway is unnecessary. Please see HDDS-6206 and HDDS-6247 for similar changes in other request/response scenarios.
Also, please add test cases in S3 acceptance tests. This helps in verifying the changes (manual testing is not scalable) and also in avoiding any regression in the future.
s3g_1 | 2022-03-05 08:19:42,809 [qtp1267149311-20] ERROR signature.AuthorizationV4HeaderParser: AWS access id shouldn't be empty. credential:/20220305/us-east-1/s3/aws4_request
s3g_1 | Mar 05, 2022 8:19:42 AM org.glassfish.jersey.internal.Errors logErrors
s3g_1 | WARNING: The following warnings have been detected: WARNING: Unknown HK2 failure detected:
s3g_1 | MultiException stack 1 of 1
s3g_1 | javax.ws.rs.WebApplicationException: The authorization header you provided is invalid.
s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.wrapOS3Exception(OzoneClientProducer.java:145)
s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getSignature(OzoneClientProducer.java:105)
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
s3g_1 | at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
s3g_1 | at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
s3g_1 | at java.base/java.lang.reflect.Method.invoke(Method.java:566)
s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:88)
s3g_1 | at org.jboss.weld.injection.StaticMethodInjectionPoint.invoke(StaticMethodInjectionPoint.java:78)
s3g_1 | at org.jboss.weld.injection.producer.ProducerMethodProducer.produce(ProducerMethodProducer.java:100)
s3g_1 | at org.jboss.weld.injection.producer.AbstractMemberProducer.produce(AbstractMemberProducer.java:161)
s3g_1 | at org.jboss.weld.bean.AbstractProducerBean.create(AbstractProducerBean.java:180)
s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785)
s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getInjectableReference(BeanManagerImpl.java:885)
s3g_1 | at org.jboss.weld.injection.FieldInjectionPoint.inject(FieldInjectionPoint.java:92)
s3g_1 | at org.jboss.weld.util.Beans.injectBoundFields(Beans.java:358)
s3g_1 | at org.jboss.weld.util.Beans.injectFieldsAndInitializers(Beans.java:369)
s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector$1.proceed(ResourceInjector.java:70)
s3g_1 | at org.jboss.weld.injection.InjectionContextImpl.run(InjectionContextImpl.java:48)
s3g_1 | at org.jboss.weld.injection.producer.ResourceInjector.inject(ResourceInjector.java:72)
s3g_1 | at org.jboss.weld.injection.producer.BasicInjectionTarget.inject(BasicInjectionTarget.java:117)
s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiComponentProvider$InjectionManagerInjectedCdiTarget.inject(CdiComponentProvider.java:874)
s3g_1 | at org.jboss.weld.bean.ManagedBean.create(ManagedBean.java:159)
s3g_1 | at org.jboss.weld.context.unbound.DependentContextImpl.get(DependentContextImpl.java:70)
s3g_1 | at org.jboss.weld.bean.ContextualInstanceStrategy$DefaultContextualInstanceStrategy.get(ContextualInstanceStrategy.java:100)
s3g_1 | at org.jboss.weld.bean.ContextualInstance.get(ContextualInstance.java:50)
s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:785)
s3g_1 | at org.jboss.weld.manager.BeanManagerImpl.getReference(BeanManagerImpl.java:808)
s3g_1 | at org.jboss.weld.util.ForwardingBeanManager.getReference(ForwardingBeanManager.java:61)
s3g_1 | at org.jboss.weld.bean.builtin.BeanManagerProxy.getReference(BeanManagerProxy.java:85)
s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.CdiUtil.getBeanReference(CdiUtil.java:127)
s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier$1.getInstance(AbstractCdiBeanSupplier.java:69)
s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.AbstractCdiBeanSupplier._provide(AbstractCdiBeanSupplier.java:103)
s3g_1 | at org.glassfish.jersey.ext.cdi1x.internal.RequestScopedCdiBeanSupplier.get(RequestScopedCdiBeanSupplier.java:46)
s3g_1 | at org.glassfish.jersey.inject.hk2.InstanceSupplierFactoryBridge.provide(InstanceSupplierFactoryBridge.java:53)
s3g_1 | at org.jvnet.hk2.internal.FactoryCreator.create(FactoryCreator.java:129)
s3g_1 | at org.jvnet.hk2.internal.SystemDescriptor.create(SystemDescriptor.java:463)
s3g_1 | at org.jvnet.hk2.internal.PerLookupContext.findOrCreate(PerLookupContext.java:46)
s3g_1 | at org.jvnet.hk2.internal.Utilities.createService(Utilities.java:2102)
s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:758)
s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.internalGetService(ServiceLocatorImpl.java:721)
s3g_1 | at org.jvnet.hk2.internal.ServiceLocatorImpl.getService(ServiceLocatorImpl.java:691)
s3g_1 | at org.glassfish.jersey.inject.hk2.AbstractHk2InjectionManager.getInstance(AbstractHk2InjectionManager.java:160)
s3g_1 | at org.glassfish.jersey.inject.hk2.ImmediateHk2InjectionManager.getInstance(ImmediateHk2InjectionManager.java:30)
s3g_1 | at org.glassfish.jersey.internal.inject.Injections.getOrCreate(Injections.java:105)
s3g_1 | at org.glassfish.jersey.server.model.MethodHandler$ClassBasedMethodHandler.getInstance(MethodHandler.java:260)
s3g_1 | at org.glassfish.jersey.server.internal.routing.PushMethodHandlerRouter.apply(PushMethodHandlerRouter.java:51)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:86)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage._apply(RoutingStage.java:89)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:69)
s3g_1 | at org.glassfish.jersey.server.internal.routing.RoutingStage.apply(RoutingStage.java:38)
s3g_1 | at org.glassfish.jersey.process.internal.Stages.process(Stages.java:173)
s3g_1 | at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:247)
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
s3g_1 | at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
s3g_1 | at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
s3g_1 | at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
s3g_1 | at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:234)
s3g_1 | at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
s3g_1 | at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:366)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:319)
s3g_1 | at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205)
s3g_1 | at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1459)
s3g_1 | at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1626)
s3g_1 | at org.apache.hadoop.ozone.s3.RootPageDisplayFilter.doFilter(RootPageDisplayFilter.java:53)
s3g_1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
s3g_1 | at org.apache.hadoop.ozone.s3.EmptyContentTypeFilter.doFilter(EmptyContentTypeFilter.java:76)
s3g_1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:201)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
s3g_1 | at org.apache.hadoop.hdds.server.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1678)
s3g_1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
s3g_1 | at org.apache.hadoop.hdds.server.http.NoCacheFilter.doFilter(NoCacheFilter.java:48)
s3g_1 | at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
s3g_1 | at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602)
s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
s3g_1 | at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
s3g_1 | at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
s3g_1 | at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
s3g_1 | at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
s3g_1 | at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
s3g_1 | at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
s3g_1 | at org.eclipse.jetty.server.Server.handle(Server.java:516)
s3g_1 | at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388)
s3g_1 | at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633)
s3g_1 | at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380)
s3g_1 | at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
s3g_1 | at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
s3g_1 | at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
s3g_1 | at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
s3g_1 | at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
s3g_1 | at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:386)
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
s3g_1 | at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
s3g_1 | at java.base/java.lang.Thread.run(Thread.java:829)
s3g_1 | Caused by: org.apache.hadoop.ozone.s3.exception.OS3Exception
s3g_1 | at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:139)
s3g_1 | at org.apache.hadoop.ozone.s3.exception.S3ErrorTable.newError(S3ErrorTable.java:126)
s3g_1 | at org.apache.hadoop.ozone.s3.signature.AuthorizationV4HeaderParser.parseCredentials(AuthorizationV4HeaderParser.java:171)
s3g_1 | at org.apache.hadoop.ozone.s3.signature.AuthorizationV4HeaderParser.parseSignature(AuthorizationV4HeaderParser.java:91)
s3g_1 | at org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor.parseSignature(AWSSignatureProcessor.java:70)
s3g_1 | at org.apache.hadoop.ozone.s3.signature.AWSSignatureProcessor$Proxy$_$$_WeldClientProxy.parseSignature(Unknown Source)
s3g_1 | at org.apache.hadoop.ozone.s3.OzoneClientProducer.getSignature(OzoneClientProducer.java:86)
s3g_1 | ... 115 more
adoroszlai
reviewed
Mar 5, 2022
| @@ -47,7 +47,13 @@ File upload and directory list | |||
| Should not contain ${result} dir1 | |||
| Should contain ${result} file | |||
|
|
|||
| List buckets with empty access id | |||
| Execute aws configure set aws_access_key_id '' | |||
| ${result} = Execute AWSS3APICli and checkrc list-buckets 255 | |||
There was a problem hiding this comment.
S3 tests are grouped, mostly based on operation type. list-buckets belongs to bucketlist.robot.
(awss3.robot is for aws s3 command, while other tests are for aws s3api.)
Also, adding it as last test case would avoid the need to run Setup s3 tests explicitly in the next test case.
There was a problem hiding this comment.
Sure, I'll move it to bucketlist.robot.
I didn't put it at last incase others adding new test would loss the accesskey.
|
For the verbose log, it's from jersey and it's print to ozone-s3g.out. |
adoroszlai
reviewed
Mar 7, 2022
Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
adoroszlai
reviewed
Mar 9, 2022
hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/OzoneClientProducer.java
Outdated
Show resolved
Hide resolved
…3/OzoneClientProducer.java Co-authored-by: Doroszlai, Attila <6454655+adoroszlai@users.noreply.github.com>
|
There may be AccessControlExceptions from OM, should we handle and send back to users? |
Which call results in |
|
A example would be the impersonate feature, as the test case of https://github.com/apache/ozone/pull/3157/files. |
Correct me if I'm wrong, but my understanding is that this would be a server-side configuration issue, to be fixed by admin, not relevant to S3 user, so should not be sent back to them. |
|
Yes, the current usage won't trigger this issue. |
adoroszlai
approved these changes
Mar 9, 2022
|
@adoroszlai Thank you for the detail review. |
adoroszlai
changed the title
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
For many errors, S3g return a fuzzy error log which are not easy for users and maintainers to find the root cause of the error.
The current error is like follows:
This ticket is to fix this issue and return more detailed messages.
What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-6419
How was this patch tested?
Test example
aws s3api --endpoint=http://localhost:9878 list-bucketsAn error occurred () when calling the ListBuckets operation:An error occurred (AuthorizationHeaderMalformed) when calling the ListBuckets operation: The authorization header you provided is invalid.