HDDS-4440- s3 performance#2259
Closed
neils-dev wants to merge 14 commits intoapache:HDDS-4440-s3-performancefrom
Closed
HDDS-4440- s3 performance#2259neils-dev wants to merge 14 commits intoapache:HDDS-4440-s3-performancefrom
neils-dev wants to merge 14 commits intoapache:HDDS-4440-s3-performancefrom
Conversation
…rt, client grpc-netty channal, server grpc-netty service. Generates grpc s3OmGrpc server protobuf and client stubs. Changes made to pom files in hadoop-common and hadoop-client-interfaces.
…PC service def for Om protocol (subtask 1 hdds-4440 - OzoneManagerServiceGrpc.java); OM Gprc server started by OM (subtask 2 hdds-4440 - GrpcOzoneManagerServer.java, OzoneManager.java); implementation fo OmTransport interface (subtask 3 hdds-4440 - GrpcOmTransport.java); implementation of OmTransportFactory for s3g gRPC (subtask 4 hdds-4440 - GrpcOmTransportFactory.java). - Necessary to disable Om ratis for this commit, set ozone.om.ratis.enable=false in ozone-site.xml; use intellij hadoop-ozone/dev-support/intellij/ozone-site.xml and intellij s3g together with invoke s3g for simple bucket commands through s3 cli.
…ent requests - fixes reported limitation on disabled om ratis configuration. S3g gRPC NOW functioning with om ratis. Added maven pom file dependency to s3gateway to fix problem with runtime not finding grpc netty ChannelFactory when run in docker cluster mode. Minor cleanup to maven pom files.
… OmRequest/OmResponse. Created OzoneClient cache in OzoneClientProducer to have single OzoneClient (GrpcOmTransport connection) process all s3 requests.
Member
|
Thanks the patch @neils-dev. Looks it has a lot of unrelated change. Is it possible that master is merged to your branch but not to the target branch? |
…ozone into HDDS-4440-s3-performance
…ions 1.38.0 and 4.1.63.Final as defined in main ozone pom build properties. Also minor modifications to GrpcServer and OzoneManager for checkstyle related fixes.
…thentication. Includes client creating OzoneToken with s3 string2sign, aws id and signature, transport through grpc (modified OzoneManagerProtocol) with request thread context. On the server side, authenticating request with s3 signature comparing with secret key stored through the delegationTokenManager. Datanode delegation token handling to follow in next commit.
…request filter to create OzoneToken for authenication and client request thread context; added changes to Om server to handle s3g grpc with ACLS through OmClientRequest; changes made to handle delegationToken based s3 object operations (put,get,list,delete requests); unit test cases added for s3g filters and aws signature; changes made to pass all current s3g unit tests.
…cation header date validator. General clean up / fixes for attempting a green build for the feature branch.
…d ACCESS_DENIED to http responses from authentication (secret key missing and signature mismatch) errors. Also added support for error propagation back to client from server through gRPC OMResponse with proper failure status and error message. Simplied error handling from http servlet UgiFilter - interpreted as fatal error at this level, s3 request errors should be handled from s3 endpoint command level and not at http filter level.
Contributor
|
@neils-dev Given that
I think we should close this mega PR and open a new one for the second sub-task. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Initial commit for s3g gRPC for command OmRequest and OmResponses. Creates:
hadoop-ozone/interface-client/pom.xml,hadoop-ozone/interface-client/target/generated-sources/protobuf/java/org/apache/hadoop/ozone/protocol/proto/OzoneManagerServiceGrpc.java- HDDS-5210hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/GrpcOzoneManagerServer.java,OzoneManager.java- HDDS-5211hadoop-ozone/interface-client/pom.xml- HDDS-5212hadoop-ozone/common/src/main/java/org/apache/hadoop/ozone/om/protocolPB/GrpcOmTransportFactory.java; dynamically selected ServiceProvider loaded class for s3ghadoop-ozone/s3gateway/src/main/resources/META-INF/services/org.apache.hadoop.ozone.om.protocolPB.OmTransportFactory- HDDS-5213Currently requires ratis disabled for Om : set(fixed/resolved) use intellijozone.om.ratis.enable=falsein ozone-site.xml;hadoop-ozone/dev-support/intellij/ozone-site.xmland intellij s3g together with invoke s3g for simple bucket commands through s3 cli.In addition, for s3g gRPC OmRequest/OmResponse a single OmTransport connection is used between s3g and the ozone manager. A cached
OzoneClientis implemented to service every s3g request.What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-5210 - (see 1. previous section)
https://issues.apache.org/jira/browse/HDDS-5211 - (see 2. previous section)
https://issues.apache.org/jira/browse/HDDS-5212 - (see 3. previous section)
https://issues.apache.org/jira/browse/HDDS-5213 - (see 4. previous section)
How was this patch tested?
Patch is initial POC for s3g gRPC OmRequest/OmResponse.
Currently requires ratis disabled for Om - (when invoking ratis server through OzoneManagerRatisServer submitRequest,createWriteRaftClientRequest(omRequest)fails when checking preconditions due to hadoop class Server.Call is null; requires attention from propagating the om SubmitRequest through the GrpcServer). (fixed/resolved)To test - use intellij ozone development cluster running through runConfigurations for invoking SCM, OM, Recon, Datanodes and S3Gateway.
Commit includes modified ozone-site.xml ((fixed/resolved) Run simple s3 file operations through aws cli:hadoop-ozone/dev-support/intellij/ozone-site.xml) to disable ratis for om (<property><name>ozone.om.ratis.enable</name><value>false</value></property>).For s3g gRPC comparison to existing s3g hadoop RPC performance measurements, a freon test is implemented,
S3BucketGenerator.java, and used.Run through intellij, deployment platform a cluster consisting of SCM, OM, Recon,Datanodes, S3Gateway:
$ ozone freon s3bg -t 2 -n 5