Simplify gateway backend registration #5111

timonwong · 2017-10-24T05:57:48Z

Description

Motivation and Context

When I'm trying to add a new gateway backend in #5103, several files need to be touched, which IMHO is not very convenient.
In this pull request, gateway backend implementations only need to focus their own scope, and
just invoke mustRegisterGatewayBackend() function in their init().

How Has This Been Tested?

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

My code follows the code style of this project.
My change requires a change to the documentation.
I have updated the documentation accordingly.
I have added tests to cover my changes.
All new and existing tests passed.

codecov · 2017-10-24T06:12:09Z

Codecov Report

Merging #5111 into master will increase coverage by 0.56%.
The diff coverage is 56.36%.

@@            Coverage Diff             @@
##           master    #5111      +/-   ##
==========================================
+ Coverage   61.25%   61.82%   +0.56%     
==========================================
  Files         193      153      -40     
  Lines       28411    26216    -2195     
==========================================
- Hits        17404    16208    -1196     
+ Misses       9742     8808     -934     
+ Partials     1265     1200      -65

Impacted Files	Coverage Δ
cmd/globals.go	`100% <ø> (ø)`	⬆️
cmd/web-handlers.go	`74.96% <33.33%> (+0.1%)`	⬆️
cmd/gateway-main.go	`19.76% <7.2%> (+7.19%)`	⬆️
cmd/gateway-gcs.go	`12.04% <74.6%> (+5.37%)`	⬆️
cmd/gateway-b2.go	`7.67% <84.78%> (+7.67%)`	⬆️
cmd/gateway-s3.go	`16.89% <84.9%> (+9.52%)`	⬆️
cmd/gateway-azure.go	`24.2% <85.18%> (+5.72%)`	⬆️
cmd/storage-rpc-client.go	`36.63% <0%> (-41.09%)`	⬇️
cmd/update-notifier.go	`58.33% <0%> (-28.34%)`	⬇️
cmd/storage-rpc-server.go	`52.63% <0%> (-26.32%)`	⬇️
... and 54 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 758d545...a187c4c. Read the comment docs.

codecov · 2017-10-24T06:12:09Z

Codecov Report

Merging #5111 into master will increase coverage by 0.46%.
The diff coverage is 54.66%.

@@            Coverage Diff             @@
##           master    #5111      +/-   ##
==========================================
+ Coverage   61.36%   61.82%   +0.46%     
==========================================
  Files         194      153      -41     
  Lines       28567    26237    -2330     
==========================================
- Hits        17529    16220    -1309     
+ Misses       9768     8821     -947     
+ Partials     1270     1196      -74

Impacted Files	Coverage Δ
cmd/globals.go	`100% <ø> (ø)`	⬆️
cmd/gateway-startup-msg.go	`90% <100%> (ø)`	⬆️
cmd/web-handlers.go	`74.96% <33.33%> (+0.1%)`	⬆️
cmd/gateway-main.go	`22.07% <45%> (+9.45%)`	⬆️
cmd/gateway-gcs.go	`11.44% <46.15%> (+4.8%)`	⬆️
cmd/gateway-s3.go	`16% <49.39%> (+8.64%)`	⬆️
cmd/gateway-azure.go	`23.57% <57.53%> (+5.12%)`	⬆️
cmd/gateway-b2.go	`6.77% <85%> (+6.77%)`	⬆️
cmd/storage-rpc-client.go	`36.63% <0%> (-41.09%)`	⬇️
cmd/update-notifier.go	`58.33% <0%> (-28.34%)`	⬇️
... and 56 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 03df692...90f0c8a. Read the comment docs.

harshavardhana

Overall a nice change, but some comments about style and structure.

harshavardhana · 2017-10-24T09:21:25Z

cmd/gateway-main.go

 	}

-	return nil, fmt.Errorf("Unrecognized backend type %s", backendType)
+	return nil, fmt.Errorf("unrecognized backend type %s", backendType)


Capitalize the strings for errors, this is to be consistent with our code base.

harshavardhana · 2017-10-24T09:22:44Z

cmd/gateway-s3.go

+`
+
+	mustRegisterGatewayBackend(gatewayBackendEntry{
+		Name: "s3",


we can use s3Backend const here..

harshavardhana · 2017-10-24T09:23:00Z

cmd/gateway-s3.go

+	mustRegisterGatewayBackend(gatewayBackendEntry{
+		Name: "s3",
+		Cmd: cli.Command{
+			Name:               "s3",


same as above and can be replicated for all gateway backends..

harshavardhana · 2017-10-24T09:24:18Z

cmd/gateway-s3.go

@@ -102,12 +161,13 @@ type s3Objects struct {
 }

 // newS3Gateway returns s3 gatewaylayer
-func newS3Gateway(host string) (GatewayLayer, error) {
+func newS3Gateway(args cli.Args) (GatewayLayer, error) {


if a gateway takes only one argument then we don't need to expand the scope here by adding more args. This refactor may have led to that. But taking in input param as host is more meaningful here.

harshavardhana · 2017-10-24T09:25:56Z

cmd/gateway-main.go

+	Name           gatewayBackend
+	Cmd            cli.Command
+	New            func(args cli.Args) (GatewayLayer, error)
+	IsExperimental bool


We don't need to call this IsExperimental just say it as Experimental is fine.

harshavardhana · 2017-10-24T09:27:25Z

cmd/gateway-main.go

 	}
+
+	gatewayBackendReg = map[gatewayBackend]gatewayBackendEntry{}


We can avoid map and make this more safer by using a slice instead?

We can avoid map and make this more safer by using a slice instead?

Not sure, would you mind telling why a slice is more safer in this scenario?

harshavardhana · 2017-10-24T09:31:39Z

cmd/gateway-main.go

-)
+// Represents the type of each item in gateway backend registry.
+type gatewayBackendEntry struct {
+	Name           gatewayBackend


there is no reason for this type if there are no constants anymore.

harshavardhana · 2017-10-24T09:34:45Z

cmd/gateway-main.go

+		return errors.New("field 'New' cannot be nil")
+	}
+
+	if e.Cmd.Action != nil {


This check is sort of incorrect with the overall usage and pattern, There is no reason we should add this check as it might be frivolous indeed. We should run a Validate() func right after the Action is registered, or just remove verifying Action.

harshavardhana · 2017-10-24T09:36:06Z

cmd/gateway-main.go

+		return fmt.Errorf("duplicate backend type found in registry: %s", string(backendType))
+	}
+
+	entry.Cmd.Action = gatewayMainAction(backendType)


This additional code here is not needed, modifying input structs causing side-affects is not a good idea usually. We should simply move this code to mustRegisterGatewayBackend() - gatewayMainAction("s3") for example.

So all you would have to do would to do is just simply do validation here and avoid mutation.

harshavardhana · 2017-10-24T09:38:10Z

cmd/gateway-azure.go

@@ -292,12 +354,13 @@ func azureParseBlockID(blockID string) (partID, subPartNumber int, uploadID, md5
 }

 // Inits azure blob storage client and returns AzureObjects.
-func newAzureLayer(host string) (GatewayLayer, error) {
+func newAzureLayer(args cli.Args) (GatewayLayer, error) {


The issue about changing input params similar to my comment on s3 gateway

balamurugana

We could have an interface approach to solve the problem. That way each gateway backend is completely independent. Below is the approach

gateway-main.go:

Add an interface

type Gateway interface{
    Name() string
    NewGatewayLayer() (GatewayLayer, error)
}

Rename gatewayMain() to startGateway()

gateway-s3.go:

In init() add subcommand by gatewayCmd.Subcommands = append(gatewayCmd.Subcommands, cli.Command{...})
Have S3Gateway compatible to Gateway interface

type S3Gateway struct {
    host string
}

func (g *S3Gateway) Name() string {
    return "s3"
}

func (g *S3Gateway) NewObjectLayer() (GatewayLayer, error) {
    // log.Println(colorYellow("\n               *** Warning: Not Ready for Production ***"))
    // have newS3Gateway() here
}

This logic needs to be done in all gateway code

timonwong · 2017-10-25T04:10:22Z

@balamurugana

I think we should not access gatewayCmd.Subcommands directly in each gateway implementation.
My ultimate goal (perhaps) is the possibility of adding "plugin" support, so we can easily opt-in/opt-out.

What about this?

type GatewayCommand interface {
	// Name is the global unique name for each gateway.
	Name() string
	// A short description of the usage of this command.
	Description() string
	// HelpTemplate returns the text template for the command help topic.
	HelpTemplate() string
	// Parse parse the command line context.
	Parse(ctx *cli.Context) error
	// NewGatewayLayer returns a new gateway layer.
	NewGatewayLayer() (GatewayLayer, error)
}

func RegisterGatewayCommand(command GatewayCommand) error {
	// We should not have multiple subcommands with same name.
	for _, c := range gatewayCmd.Subcommands {
		if c.Name == command.Name() {
			return fmt.Errorf("duplicate gateway command: %s", command.Name())
		}
	}

	gatewayCmd.Subcommands = append(gatewayCmd.Subcommands, cli.Command{
		Name:               command.Name(),
		Usage:              command.Description(),
		Action:             gatewayCommandMain(command),
		CustomHelpTemplate: command.HelpTemplate(),
		Flags:              append(serverFlags, globalFlags...),
		HideHelpCommand:    true,
	})
	return nil
}

balamurugana · 2017-10-25T04:57:38Z

@timonwong This approach brings a cyclic usage i.e. gateway-main.go calls a method on registration. To make things simple, you can have a registerCommand() in gateway-main.go and every gateway layer calls this function for its registration for cli package. This way each knowledge is abstracted within each gateway layer.

timonwong · 2017-10-27T03:05:26Z

Can you have a look latest commit, though I'm afraid it's not very feasible, because:

RegisterGatewayCommand() requires a GatewayCommandDesc with command line usage, help, and a function to parse command line and create a Gateway. (Because I don't want to call gatewayMain in each gateway implementation)
gateway-main.go handles actual command action, then call Gateway:NewObjectLayer() to fully bring up a gateway layer.

balamurugana · 2017-10-27T03:50:34Z

cmd/gateway-main.go

-	// Add more backends here.
-)
+// GatewayCommandDesc is descriptor for gateway command.
+type GatewayCommandDesc struct {


Other than cli package, this information is nowhere required. You could directly pass cli.Command{} from each gateway to MustRegisterGatewayCommand() and RegisterGatewayCommand().

balamurugana · 2017-10-27T03:54:53Z

Can you have a look latest commit, though I'm afraid it's not very feasible, because:

RegisterGatewayCommand() requires a GatewayCommandDesc with command line usage, help, and a function to parse command line and create a Gateway. (Because I don't want to call gatewayMain in each gateway implementation)

GatewayCommandDesc is not required. See my above comment.
gatewayMain() is a common code to know how to start a gateway. Calling startGateway() in each gateway is correct

gateway-main.go handles actual command action, then call Gateway:NewObjectLayer() to fully bring up a gateway layer.

Note that all gateway-main and gateway-*.go are same package. Its not required to visualize as separate layer.

timonwong · 2017-10-27T08:02:56Z

@balamurugana Done, just check the latest commit

balamurugana

LGTM

In minio#5111, 'help' subcommand was moved to `startGateway` during review, but not get restored then. So the "help" subcommand won't function if credentials are not provided.

timonwong force-pushed the cleanup-gateway branch from 4dd4004 to a187c4c Compare October 24, 2017 06:12

timonwong force-pushed the cleanup-gateway branch 3 times, most recently from 9272afa to 47b8a77 Compare October 24, 2017 08:33

nitisht requested review from harshavardhana and balamurugana October 24, 2017 09:03

harshavardhana requested changes Oct 24, 2017

View reviewed changes

timonwong force-pushed the cleanup-gateway branch 2 times, most recently from 913aabe to 5c75965 Compare October 24, 2017 12:21

balamurugana requested changes Oct 25, 2017

View reviewed changes

timonwong force-pushed the cleanup-gateway branch 4 times, most recently from f9e0147 to 5110625 Compare October 25, 2017 12:35

deekoder force-pushed the master branch from 1f02c70 to db3fed2 Compare October 25, 2017 18:02

timonwong force-pushed the cleanup-gateway branch 3 times, most recently from 42632c9 to 6becd79 Compare October 26, 2017 09:58

balamurugana reviewed Oct 27, 2017

View reviewed changes

timonwong force-pushed the cleanup-gateway branch from 6becd79 to a147717 Compare October 27, 2017 05:53

Simplify gateway backend registration

90f0c8a

timonwong force-pushed the cleanup-gateway branch 3 times, most recently from 4970069 to 90f0c8a Compare October 27, 2017 06:47

harshavardhana approved these changes Oct 27, 2017

View reviewed changes

balamurugana approved these changes Oct 27, 2017

View reviewed changes

deekoder added the do-not-close label Oct 27, 2017

deekoder merged commit 6400f50 into minio:master Oct 27, 2017

timonwong mentioned this pull request Oct 28, 2017

Fix help subcommand for each minio gateway #5128

Closed

8 tasks

dvstate pushed a commit to dvstate/minio that referenced this pull request Nov 2, 2017

Simplify gateway backend registration (minio#5111)

5452f71

timonwong deleted the cleanup-gateway branch November 2, 2017 15:37

harshavardhana removed the do-not-close label Nov 16, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Simplify gateway backend registration #5111

Simplify gateway backend registration #5111

timonwong commented Oct 24, 2017 •

edited

Loading

codecov bot commented Oct 24, 2017

codecov bot commented Oct 24, 2017 •

edited

Loading

harshavardhana left a comment

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

timonwong Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

harshavardhana Oct 24, 2017

balamurugana left a comment •

edited

Loading

timonwong commented Oct 25, 2017

balamurugana commented Oct 25, 2017

timonwong commented Oct 27, 2017

balamurugana Oct 27, 2017

balamurugana commented Oct 27, 2017

timonwong commented Oct 27, 2017

balamurugana left a comment

		}

		gatewayBackendReg = map[gatewayBackend]gatewayBackendEntry{}

Simplify gateway backend registration #5111

Simplify gateway backend registration #5111

Conversation

timonwong commented Oct 24, 2017 • edited Loading

Description

Motivation and Context

How Has This Been Tested?

Types of changes

Checklist:

codecov bot commented Oct 24, 2017

Codecov Report

codecov bot commented Oct 24, 2017 • edited Loading

Codecov Report

harshavardhana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

balamurugana left a comment • edited Loading

Choose a reason for hiding this comment

timonwong commented Oct 25, 2017

balamurugana commented Oct 25, 2017

timonwong commented Oct 27, 2017

Choose a reason for hiding this comment

balamurugana commented Oct 27, 2017

timonwong commented Oct 27, 2017

balamurugana left a comment

Choose a reason for hiding this comment

timonwong commented Oct 24, 2017 •

edited

Loading

codecov bot commented Oct 24, 2017 •

edited

Loading

balamurugana left a comment •

edited

Loading