Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Beta #1

Merged
merged 30 commits into from
Jul 6, 2017
Merged

Beta #1

merged 30 commits into from
Jul 6, 2017

Conversation

xlr-8
Copy link
Contributor

@xlr-8 xlr-8 commented Jun 15, 2017

No description provided.

Hugo Rosnet added 14 commits June 8, 2017 17:09
This connector sets the fundations for this library.

It needs access/secret keys, regions and eventually an aws.Config to be
created. If no aws.Config is provided, then a default one will be used.

Among the thing it does:
 * It understands 'complex' list of regions - with regex. Therefore 'eu-*'
and 'us-*' can both be passed to have all those regions available.
 * It saves the accountID for later use - such as AMI or Volume snapshots.
 * It instanciates sessions for later use of services: ELB, EC2, etc.

It should provide an abstraction easy enough to use all the services of
AWS in multiple regions without having to bother with their multiple
packages, connectors, etc.
This methods simply gets all the ELB (v1) from all the regions that have
been enabled in the connector.
In order to use the library, only a couple parameters need to be set, in
fact the library needs as little as 1 call to be ready to be used.

A first one to get a connector object, and the other calls to directly
gather information from their AWS account/regions.
In case invalid (empty list, or non-matching pattern) regions the code
will now return an error with the proper reason.
The comments have been added in case we want to generate documentation
via godoc, like this is the case on many other public projects.

The const variable didn't need to be accessed by all the methods so
moving it to the function using it was enough.
A new type of error has been created in order to save multiple errors
while doing call to the API. This was done to both be able to save extra
information (all errors, region & servicename), and to return the
standard error type.
Those methods will allow to fetch:
* Instances
* VPCs
* Volumes
* Snapshots (only belonging to the accountID)
* AMI (only belonging to the accountID)
* Subnet
* Security groups
In AWS SDK-go, the elbv2 represent the ELB of second generation, which
are basically the ALB, but those live in another package than the first
version.
In order to mock AWS calls, we have to use interfaces, otherwise we
wouldn't be able to create our own methods to fake them.

The modification has also been followed by the update of:
svc.$SERVICE.ServiceName to $SERVICE.ServiceName

As the interface doesn't have this variable, and this variable is in
fact set as a const in the package, so there is no need to try to access
the variable from each service.
This addition allows to fetch DBInstances as well as Tags associated
with RDS resources.
The implicit convention in Go wants to have the exported methods/struct
at the top of the file, while the non-exported ones at the bottom.
The methods added allow to fetch Clusters as well as tags from
elasticache resources - same model as RDS.

Tests have also been added with mocking of the AWS API calls, however
the method 'New' couldn't be mocked, and it didn't make too much sense
to mock it anyway, therefore this part is missing.
GoDoc relies on comments above struct and methods, so simple
explanations have been added to every exported element.
An extra test has been added to elasticache, as 'multiple regions with
error' was missing for the tags.

The formatting of test messages has been revised in order to comply
with: `$test-name [$index] - $element: received=%+v | expected=%+v`

```
-- FAIL: TestGetElasticacheTags (0.00s)
	elasticache_test.go:354: multiple region with error [3] - tags: received=[{

		} {
		  TagList: [{
		      Key: "test",
		      Value: "2"
		    }]
		}] | expected=[{

		} {
		  TagList: [{
		      Key: "test",
		      Value: "3"
		    }]
		}]
FAIL
```

The method checkErrors has also been moved to another file, as it will
be reused for other tests.
core/error.go Outdated

// RawsErr type satisfies the standard error interface, thus allowing us to return an error when doing multiple call via
// the go-SDK, even though multiple errors are met; which is why APIErrs save those more specific errors.
type RawsErr struct {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will make a type of batch errors like this

type Errs []Err

func (errs Errs) Error() string {
  	var output [][]string
  
  	for _, e := range errs {
  		output = append(output, []string{e.Region(), e.Service()})
  	}
  	return fmt.Sprintf("%d error(s) occured: %s", len(errs), output)
  }

Then we don't need the AppendError we could just do

var errs Errs

errs = append(errs, NewAPIError(e, "the region", "the service"))

Having this, we can return Errs as nil without having to handle if the slice have values or not

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, good idea - will apply

core/elbv2.go Outdated
)

// Returns a list of ELB (v2) - also known as ALB - based on the input from the different regions
func (c *Connector) GetLoadBalancersV2(input *elbv2.DescribeLoadBalancersInput) ([]*elbv2.DescribeLoadBalancersOutput, error) {
Copy link
Contributor

@ifraixedes ifraixedes Jun 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because of https://github.com/cycloidio/raws/pull/1/files#r122167049

I would return and Errs type rather than an error interface, it isn't a problem for the caller because Errs satisfies the error interface

This comment applies to all the functions that the package has now and they return RawsErr type as error interface

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But as Errs satisfies error, why returning it, rather than an error type? As we discussed, my understanding was that it was more standard, and would simply make more sense. No?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ups, I didn't want to mean that.

For such reason, I was complaining about AWS SDK returns a generic error than their specific error interfaces on methods that they can only return one specific error type. If their function can return multiple interface type of errors in a function, you have to choose the narrowest interface which is satisfied by all of those, obviously the error interface should be satisfied by all of them, so there is always that chance.

I would like to remember the right terminology but if I don't remember bad (I'm shit remember names and terminology), a function which is more extensible must be covariance in their arguments and contravariance in their return values (Covariance and contravariance (computer science)

What's standard is to return an value which satisfies the error interface. What I wanted to mean is that returning type struct rather than a type interface has more limitations for the callers and from the creator point of view, could lead to export some struct fields that they can be removed later on, but the callers are relying on them, so it's more prone to break compatibility than using interfaces.

For this reason, I suggest to return the interface.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I took all notes into considerations!

Hugo Rosnet added 3 commits June 15, 2017 15:40
The `RawsErr` did not make too much sense as we could simple defined a
`type Errs []Err`, thus simplifying the adding of errors as well as the
overall management/return.

The naming of errors in methods has been reviewed to be `errs`
everywhere, and those methods now check whether an error happened or
not before adding it (same for resources), in order to have a data more
representative of what really happens during calls.
The .idea repository is created by my IDE, it does not make sense to try
tracking this file, as it is useless for the project.
The method is fetching DB Instances, so it doesn't make sense to store
those as 'elb' while really they are 'instance'.
)

func checkErrors(t *testing.T, name string, index int, err error, expected error) {
func checkErrors(t *testing.T, name string, index int, err Errs, expected Errs) {
Copy link
Contributor

@ifraixedes ifraixedes Jun 15, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because it's used reflect.DeepEqual the parameters wouldn't have needed to be changed and the function will be more generic for any test. Being a test we don't need to show any semantics in docs which types can provide

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I simply thought, that as we were using our own type, it would make more sense to use it there as well. But that would indeed work with both, I'll revert it.

core/error.go Outdated
}

// Error returns a string containing the region, service as well as the original API error message.
func (e *callErr) Error() string {
func (e callErr) Error() string {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason behind this change? (With this change I mean the change in the type which is declared the methods required by the interface)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hum, it's a mistake from my side, as when using Errs I couldn't have a pointer receiver for Error() otherwise I would hit: Errs does not implement error (Error method has pointer receiver); and I wrongfully applied it to callErr as well.

Hugo Rosnet added 2 commits June 15, 2017 16:22
The receiver for callErr did not need to be updated.

The checkError didn't need to be updated as both Err implement error, we
can keep this generic type to test other things.
These methods allow to get buckets/objects and their tags in multiple
regions. One improvement could be to provide a region for GetObjects
otherwise the Connector, will try to look it up through the regions.

That could be seen as both convenient and 'unecessary' as extra call are
done, but the client should know which regions they manage afterall.
Hugo Rosnet added 2 commits June 16, 2017 10:46
Basic tests have been added to the various components currently used in
the project (ec2, elb, elbv2, s3 and rds). The mocking is fairly simple,
as the goal is to ensure that our methods are actually returning the
good set of data and/or errors.

Tests regarding accountID, and such things haven't been added as the
purpose is not to reproduce AWS behaviour - we trust them -, but simply
what our methods are doing.
Hugo Rosnet added 2 commits June 19, 2017 18:29
In order to make testing/resusability easier, the AWS component used to
create a Connector are being isolated into 'configureAWS' function.
If we realise that we need to store them, we could also easily fix the
code.
The connector file is a bit trickier to test/mock because there are
various calls to AWS which can't really be tested/mocked ('New'), but
because they have been gathered in one function, the rest had be tested.
Hugo Rosnet added 7 commits June 20, 2017 09:38
The error declares couple struct/interface/methods which needed to be
tested as well, to validate the different output we would have by
calling them.
The package was located in a subdir, which didn't make much sense for a
library like this one. Leaving everything at the root is more
go-idomatic.
In order to provide a simple example the main.go has been moved to an
'example' directoy, rather than being left in the root, and give the
impression that it is a binary.
In order to have a small CI running on the library we needed to add
those elements.
The default 'install' rule of travis-ci is 'true' when a Makefile is
present, so in order to avoid that, it will now download the packages
for the project and tests.
It is meant to start explaining what is the project, and how to use the
library, as well as the current limitations or elements to keep in mind.
The README had one or two typos in the getting started, and could
benefit from links for both example and license file.
@coveralls
Copy link

coveralls commented Jul 5, 2017

Coverage Status

Changes Unknown when pulling 0cdf330 on beta into ** on master**.

@xescugc
Copy link
Member

xescugc commented Jul 5, 2017

Overall I see no errors, just some punctuations (it's a big PR hehe)

For the documentation I recomend you to read this post which explains the go way.

Notice this comment is a complete sentence that begins with the name of the element it describes. This important convention allows us to generate documentation in a variety of formats, from plain text to HTML to UNIX man pages, and makes it read better when tools truncate it for brevity, such as when they extract the first line or sentence.

For the code everithing is clear (I read it in an overview as its big) but I think that it's well formatted, ther are some funcions in which you could use a go syntax suggar on declare repeated types on the arguments, example you could do:

   func NewConnector(accessKey, secretKey string, regions []string, config *aws.Config) (*Connector, error) { ... }

instead of

  func NewConnector(accessKey string, secretKey string, regions []string, config *aws.Config) (*Connector, error) {

Other things are opiniated, the use of var instead of := everyone has an opinion on that :)

@cycloidio cycloidio deleted a comment from coveralls Jul 6, 2017
@xlr-8
Copy link
Contributor Author

xlr-8 commented Jul 6, 2017

Thank you for the comments, I'll take a look at the doc link.

Regarding the notation, for me it really depends of the situation.
When the variables are just declared, and used later on, then it makes sense to me to do so, I find it also more explicit when using those in an example. However for directly used/initialized variable within the code, then I don't see the problem with that, as it's self-explicit.

And on the arguments, I personally prefer to use the shortened version, only when all arguments have the same type, I find it confusing otherwise.
But as you said, everyone has his opinion :)

@xlr-8 xlr-8 mentioned this pull request Jul 6, 2017
@xlr-8 xlr-8 requested a review from xescugc July 6, 2017 09:28
@xlr-8 xlr-8 merged commit f5d0e96 into master Jul 6, 2017
@xlr-8 xlr-8 deleted the beta branch July 6, 2017 09:33
@ifraixedes
Copy link
Contributor

I also prefer to use the long syntax to declare variables var ... and only to use the short syntax for loops (for i := ...) and conditionals (if err := ) because

  1. Go doesn't compile if you use var on loops on conditionals e.g. for var i = ...
  2. Even though it would compile, I'm not sure if having the var word after for and if could be quite unreadable.

Obviously 1 enforce to use the short syntax on such cases if you only want to declare the variable inside of the loop and conditional scope

Regarding the Go syntax sugar for the types of the function arguments, I wold have prefer that it won't exist, so I prefer to use the long syntax, rather than the short, although it's something that, at the beginning I used the short syntax, later I've been doubting between both and now I prefer the long one.

The case that @xlr-8 mentions could be a good case to consider to use it, however I'm still not sold on using both rather than one.

As you commented this is a personal opinion, so I don't think that we should enforce one, they are minor things to remember and because Go has a short set of instructions and statements and syntax sugar tricks, every Go developer know them, so the syntax looks familiar. If you would like to enforce one, I'm not agains but only if we can detect them through a linter.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants