Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Kinesis logging endpoint #177

Merged
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ require (
github.com/alecthomas/units v0.0.0-20190924025748-f65c72e2690d // indirect
github.com/blang/semver v3.5.1+incompatible
github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0
github.com/fastly/go-fastly/v2 v2.0.0
github.com/fastly/go-fastly/v2 v2.1.0
github.com/fastly/kingpin v2.1.12-0.20191105091915-95d230a53780+incompatible
github.com/fatih/color v1.7.0
github.com/frankban/quicktest v1.5.0 // indirect
Expand Down
4 changes: 2 additions & 2 deletions go.sum
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0 h1:9
github.com/dustinkirkland/golang-petname v0.0.0-20191129215211-8e5a1ed0cff0/go.mod h1:V+Qd57rJe8gd4eiGzZyg4h54VLHmYVVw54iMnlAMrF8=
github.com/emirpasic/gods v1.12.0 h1:QAUIPSaCu4G+POclxeqb3F+WPpdKqFGlw36+yOzGlrg=
github.com/emirpasic/gods v1.12.0/go.mod h1:YfzfFFoVP/catgzJb4IKIqXjX78Ha8FMSDh3ymbK86o=
github.com/fastly/go-fastly/v2 v2.0.0 h1:dCPOEbTXy8ic5CGj6YibPNvRYG01y15H7kHTlC4d57k=
github.com/fastly/go-fastly/v2 v2.0.0/go.mod h1:+gom+YR+9Q5I4biSk/ZjHQGWXxqpRxC3YDVYQcRpZwQ=
github.com/fastly/go-fastly/v2 v2.1.0 h1:VQWCFiUZyCcknyxyhiQwU0MjLYzeK4g1mLo/US2r/P0=
github.com/fastly/go-fastly/v2 v2.1.0/go.mod h1:+gom+YR+9Q5I4biSk/ZjHQGWXxqpRxC3YDVYQcRpZwQ=
github.com/fastly/kingpin v2.1.12-0.20191105091915-95d230a53780+incompatible h1:FhrXlfhgGCS+uc6YwyiFUt04alnjpoX7vgDKJxS6Qbk=
github.com/fastly/kingpin v2.1.12-0.20191105091915-95d230a53780+incompatible/go.mod h1:U8UynVoU1SQaqD2I4ZqgYd5lx3A1ipQYn4aSt2Y5h6c=
github.com/fatih/color v1.7.0 h1:DkWD4oS2D8LGGgTQ6IvwJJXSL5Vp2ffcQg58nFV38Ys=
Expand Down
6 changes: 6 additions & 0 deletions pkg/api/interface.go
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,12 @@ type Interface interface {
UpdateS3(*fastly.UpdateS3Input) (*fastly.S3, error)
DeleteS3(*fastly.DeleteS3Input) error

CreateKinesis(*fastly.CreateKinesisInput) (*fastly.Kinesis, error)
ListKineses(*fastly.ListKinesesInput) ([]*fastly.Kinesis, error)
GetKinesis(*fastly.GetKinesisInput) (*fastly.Kinesis, error)
UpdateKinesis(*fastly.UpdateKinesisInput) (*fastly.Kinesis, error)
DeleteKinesis(*fastly.DeleteKinesisInput) error

CreateSyslog(*fastly.CreateSyslogInput) (*fastly.Syslog, error)
ListSyslogs(*fastly.ListSyslogsInput) ([]*fastly.Syslog, error)
GetSyslog(*fastly.GetSyslogInput) (*fastly.Syslog, error)
Expand Down
15 changes: 15 additions & 0 deletions pkg/app/run.go
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,7 @@ import (
"github.com/fastly/cli/pkg/logging/honeycomb"
"github.com/fastly/cli/pkg/logging/https"
"github.com/fastly/cli/pkg/logging/kafka"
"github.com/fastly/cli/pkg/logging/kinesis"
"github.com/fastly/cli/pkg/logging/logentries"
"github.com/fastly/cli/pkg/logging/loggly"
"github.com/fastly/cli/pkg/logging/logshuttle"
Expand Down Expand Up @@ -189,6 +190,13 @@ func Run(args []string, env config.Environment, file config.File, configFilePath
s3Update := s3.NewUpdateCommand(s3Root.CmdClause, &globals)
s3Delete := s3.NewDeleteCommand(s3Root.CmdClause, &globals)

kinesisRoot := kinesis.NewRootCommand(loggingRoot.CmdClause, &globals)
kinesisCreate := kinesis.NewCreateCommand(kinesisRoot.CmdClause, &globals)
kinesisList := kinesis.NewListCommand(kinesisRoot.CmdClause, &globals)
kinesisDescribe := kinesis.NewDescribeCommand(kinesisRoot.CmdClause, &globals)
kinesisUpdate := kinesis.NewUpdateCommand(kinesisRoot.CmdClause, &globals)
kinesisDelete := kinesis.NewDeleteCommand(kinesisRoot.CmdClause, &globals)

syslogRoot := syslog.NewRootCommand(loggingRoot.CmdClause, &globals)
syslogCreate := syslog.NewCreateCommand(syslogRoot.CmdClause, &globals)
syslogList := syslog.NewListCommand(syslogRoot.CmdClause, &globals)
Expand Down Expand Up @@ -429,6 +437,13 @@ func Run(args []string, env config.Environment, file config.File, configFilePath
s3Update,
s3Delete,

kinesisRoot,
kinesisCreate,
kinesisList,
kinesisDescribe,
kinesisUpdate,
kinesisDelete,

syslogRoot,
syslogCreate,
syslogList,
Expand Down
75 changes: 75 additions & 0 deletions pkg/app/run_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -859,6 +859,81 @@ COMMANDS
--version=VERSION Number of service version
-n, --name=NAME The name of the S3 logging object

logging kinesis create --name=NAME --version=VERSION --stream-name=STREAM-NAME --access-key=ACCESS-KEY --secret-key=SECRET-KEY --region=REGION [<flags>]
Create an Amazon Kinesis logging endpoint on a Fastly service version

-n, --name=NAME The name of the Kinesis logging object. Used
as a primary key for API access
--version=VERSION Number of service version
Integralist marked this conversation as resolved.
Show resolved Hide resolved
--stream-name=STREAM-NAME The Amazon Kinesis stream to send logs to
--access-key=ACCESS-KEY The access key associated with the target
Amazon Kinesis stream
--secret-key=SECRET-KEY The secret key associated with the target
Amazon Kinesis stream
--region=REGION The AWS region where the Kinesis stream
exists
-s, --service-id=SERVICE-ID Service ID
--format=FORMAT Apache style log formatting
--format-version=FORMAT-VERSION
The version of the custom logging format used
for the configured endpoint. Can be either 2
(default) or 1
--response-condition=RESPONSE-CONDITION
The name of an existing condition in the
configured endpoint, or leave blank to always
execute
--placement=PLACEMENT Where in the generated VCL the logging call
should be placed, overriding any
format_version default. Can be none or
waf_debug

logging kinesis list --version=VERSION [<flags>]
List Kinesis endpoints on a Fastly service version

-s, --service-id=SERVICE-ID Service ID
--version=VERSION Number of service version

logging kinesis describe --version=VERSION --name=NAME [<flags>]
Show detailed information about a Kinesis logging endpoint on a Fastly
service version

-s, --service-id=SERVICE-ID Service ID
--version=VERSION Number of service version
-n, --name=NAME The name of the Kinesis logging object

logging kinesis update --version=VERSION --name=NAME [<flags>]
Update a Kinesis logging endpoint on a Fastly service version

--version=VERSION Number of service version
-n, --name=NAME The name of the Kinesis logging object
-s, --service-id=SERVICE-ID Service ID
--new-name=NEW-NAME New name of the Kinesis logging object
--stream-name=STREAM-NAME Your Kinesis stream name
--access-key=ACCESS-KEY Your Kinesis account access key
--secret-key=SECRET-KEY Your Kinesis account secret key
--region=REGION The AWS region where the Kinesis stream
exists
--format=FORMAT Apache style log formatting
--format-version=FORMAT-VERSION
The version of the custom logging format used
for the configured endpoint. Can be either 2
(default) or 1
--response-condition=RESPONSE-CONDITION
The name of an existing condition in the
configured endpoint, or leave blank to always
execute
--placement=PLACEMENT Where in the generated VCL the logging call
should be placed, overriding any
format_version default. Can be none or
waf_debug

logging kinesis delete --version=VERSION --name=NAME [<flags>]
Delete a Kinesis logging endpoint on a Fastly service version

--version=VERSION Number of service version
-n, --name=NAME The name of the Kinesis logging object
-s, --service-id=SERVICE-ID Service ID

logging syslog create --name=NAME --version=VERSION --address=ADDRESS [<flags>]
Create a Syslog logging endpoint on a Fastly service version

Expand Down
109 changes: 109 additions & 0 deletions pkg/logging/kinesis/create.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
package kinesis

import (
"io"

"github.com/fastly/cli/pkg/common"
"github.com/fastly/cli/pkg/compute/manifest"
"github.com/fastly/cli/pkg/config"
"github.com/fastly/cli/pkg/errors"
"github.com/fastly/cli/pkg/text"
"github.com/fastly/go-fastly/v2/fastly"
)

// CreateCommand calls the Fastly API to create an Amazon Kinesis logging endpoint.
type CreateCommand struct {
common.Base
manifest manifest.Data

// required
EndpointName string // Can't shadow common.Base method Name().
Version int
StreamName string
AccessKey string
SecretKey string
Region string

// optional
Format common.OptionalString
FormatVersion common.OptionalUint
ResponseCondition common.OptionalString
Placement common.OptionalString
}

// NewCreateCommand returns a usable command registered under the parent.
func NewCreateCommand(parent common.Registerer, globals *config.Data) *CreateCommand {
var c CreateCommand
c.Globals = globals
c.manifest.File.Read(manifest.Filename)
c.CmdClause = parent.Command("create", "Create an Amazon Kinesis logging endpoint on a Fastly service version").Alias("add")

// required
c.CmdClause.Flag("name", "The name of the Kinesis logging object. Used as a primary key for API access").Short('n').Required().StringVar(&c.EndpointName)
c.CmdClause.Flag("version", "Number of service version").Required().IntVar(&c.Version)
c.CmdClause.Flag("stream-name", "The Amazon Kinesis stream to send logs to").Required().StringVar(&c.StreamName)
c.CmdClause.Flag("access-key", "The access key associated with the target Amazon Kinesis stream").Required().StringVar(&c.AccessKey)
c.CmdClause.Flag("secret-key", "The secret key associated with the target Amazon Kinesis stream").Required().StringVar(&c.SecretKey)
c.CmdClause.Flag("region", "The AWS region where the Kinesis stream exists").Required().StringVar(&c.Region)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are we sure 'region' is required? It doesn't look to be in the API docs example (nor in Go-Fastly).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, it is required. The configuration validation will fail if it is not present. I see what you mean in the case of go-fastly. I followed the existing patterns of other logging endpoints when I did that work and they all seem to only have ServiceId and ServiceVersion as required. My guess is that the enforcement of required fields was deferred to the config validation, but maybe we want to revisit that.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question:

The configuration validation will fail if it is not present.

I presume 'config validation' refers to the upstream API's own validation, would that be correct?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see Go-Fastly currently doesn't define validation for the 'region' field, but if we're saying the API itself is expecting a region field otherwise it'll return an error, then we should ensure a validation check for the 'region' field is added into Go-Fastly (I have an existing PR open I can add this to).

But for the time being, this is fine to be marked as .Required() considering Go-Fastly doesn't currently have logic to catch the missing field.

Note: I can't actually test whether the API expects 'region' to be provided because it seems Kinesis logging isn't available on my personal Fastly account 😬 I did login as an admin to see if it's something I could enable as a feature flag but nothing obvious was coming up in my search.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, just FYI (most of which will be obvious to you): my perspective on this is that we have three tools at play here:

CLI > Go-Fastly > API

We should be able to trust the API to always have the right validation in place, but (as you know) by having validation outside of the API it means we can prevent slowing down a client trying to carry out operations that otherwise we know are just going to fail by the time they reach the API layer, so short-circuiting those requests is better for the client as far as the 'feedback loop' is concerned but also good for the API because it doesn't have to handle the request in the first place.

Considering Go-Fastly and CLI sit at effectively the same level (i.e. they exist on the client) we need to decide which 'tool' is responsible for handling validation of obvious issues with a client request, and for me that should be Go-Fastly as it's the thing that's ultimately fronting the API.

So my hope is that with the work @doramatadora has been doing on the new OpenAPI specification it will mean we can move to a place where we more concretely know what is a required field and have Go-Fastly handle validation and thus reduce duplicating logic in the CLI (although admittedly at this point in time the OpenAPI spec for Kinesis logging still doesn't indicate that 'region' is a required field nor does it show it as part of its example code so that would likely need updating).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I presume 'config validation' refers to the upstream API's own validation, would that be correct?

That's correct.

Note: I can't actually test whether the API expects 'region' to be provided because it seems Kinesis logging isn't available on my personal Fastly account grimacing I did login as an admin to see if it's something I could enable as a feature flag but nothing obvious was coming up in my search.

If you send me the service id in slack I can enable this for you.

I'll also follow-up on the docs side of things and open a change request if needed to have it indicate region as required.


// optional
c.CmdClause.Flag("service-id", "Service ID").Short('s').StringVar(&c.manifest.Flag.ServiceID)
c.CmdClause.Flag("format", "Apache style log formatting").Action(c.Format.Set).StringVar(&c.Format.Value)
c.CmdClause.Flag("format-version", "The version of the custom logging format used for the configured endpoint. Can be either 2 (default) or 1").Action(c.FormatVersion.Set).UintVar(&c.FormatVersion.Value)
c.CmdClause.Flag("response-condition", "The name of an existing condition in the configured endpoint, or leave blank to always execute").Action(c.ResponseCondition.Set).StringVar(&c.ResponseCondition.Value)
c.CmdClause.Flag("placement", "Where in the generated VCL the logging call should be placed, overriding any format_version default. Can be none or waf_debug").Action(c.Placement.Set).StringVar(&c.Placement.Value)

return &c
}

// createInput transforms values parsed from CLI flags into an object to be used by the API client library.
func (c *CreateCommand) createInput() (*fastly.CreateKinesisInput, error) {
var input fastly.CreateKinesisInput

serviceID, source := c.manifest.ServiceID()
if source == manifest.SourceUndefined {
return nil, errors.ErrNoServiceID
}

input.ServiceID = serviceID
input.ServiceVersion = c.Version
input.Name = c.EndpointName
input.StreamName = c.StreamName
input.AccessKey = c.AccessKey
input.SecretKey = c.SecretKey
input.Region = c.Region

if c.Format.WasSet {
input.Format = c.Format.Value
}

if c.FormatVersion.WasSet {
input.FormatVersion = c.FormatVersion.Value
}

if c.ResponseCondition.WasSet {
input.ResponseCondition = c.ResponseCondition.Value
}

if c.Placement.WasSet {
input.Placement = c.Placement.Value
}

return &input, nil
}

// Exec invokes the application logic for the command.
func (c *CreateCommand) Exec(in io.Reader, out io.Writer) error {
input, err := c.createInput()
if err != nil {
return err
}

d, err := c.Globals.Client.CreateKinesis(input)
if err != nil {
return err
}

text.Success(out, "Created Kinesis logging endpoint %s (service %s version %d)", d.Name, d.ServiceID, d.ServiceVersion)
return nil
}
48 changes: 48 additions & 0 deletions pkg/logging/kinesis/delete.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
package kinesis

import (
"io"

"github.com/fastly/cli/pkg/common"
"github.com/fastly/cli/pkg/compute/manifest"
"github.com/fastly/cli/pkg/config"
"github.com/fastly/cli/pkg/errors"
"github.com/fastly/cli/pkg/text"
"github.com/fastly/go-fastly/v2/fastly"
)

// DeleteCommand calls the Fastly API to delete an Amazon Kinesis logging endpoint.
type DeleteCommand struct {
common.Base
manifest manifest.Data
Input fastly.DeleteKinesisInput
}

// NewDeleteCommand returns a usable command registered under the parent.
func NewDeleteCommand(parent common.Registerer, globals *config.Data) *DeleteCommand {
var c DeleteCommand
c.Globals = globals
c.manifest.File.Read(manifest.Filename)
c.CmdClause = parent.Command("delete", "Delete a Kinesis logging endpoint on a Fastly service version").Alias("remove")
c.CmdClause.Flag("version", "Number of service version").Required().IntVar(&c.Input.ServiceVersion)
c.CmdClause.Flag("name", "The name of the Kinesis logging object").Short('n').Required().StringVar(&c.Input.Name)
c.CmdClause.Flag("service-id", "Service ID").Short('s').StringVar(&c.manifest.Flag.ServiceID)

return &c
}

// Exec invokes the application logic for the command.
func (c *DeleteCommand) Exec(in io.Reader, out io.Writer) error {
serviceID, source := c.manifest.ServiceID()
if source == manifest.SourceUndefined {
return errors.ErrNoServiceID
}
c.Input.ServiceID = serviceID

if err := c.Globals.Client.DeleteKinesis(&c.Input); err != nil {
return err
}

text.Success(out, "Deleted Kinesis logging endpoint %s (service %s version %d)", c.Input.Name, c.Input.ServiceID, c.Input.ServiceVersion)
return nil
}
59 changes: 59 additions & 0 deletions pkg/logging/kinesis/describe.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
package kinesis

import (
"fmt"
"io"

"github.com/fastly/cli/pkg/common"
"github.com/fastly/cli/pkg/compute/manifest"
"github.com/fastly/cli/pkg/config"
"github.com/fastly/cli/pkg/errors"
"github.com/fastly/go-fastly/v2/fastly"
)

// DescribeCommand calls the Fastly API to describe an Amazon Kinesis logging endpoint.
type DescribeCommand struct {
common.Base
manifest manifest.Data
Input fastly.GetKinesisInput
}

// NewDescribeCommand returns a usable command registered under the parent.
func NewDescribeCommand(parent common.Registerer, globals *config.Data) *DescribeCommand {
var c DescribeCommand
c.Globals = globals
c.manifest.File.Read(manifest.Filename)
c.CmdClause = parent.Command("describe", "Show detailed information about a Kinesis logging endpoint on a Fastly service version").Alias("get")
c.CmdClause.Flag("service-id", "Service ID").Short('s').StringVar(&c.manifest.Flag.ServiceID)
c.CmdClause.Flag("version", "Number of service version").Required().IntVar(&c.Input.ServiceVersion)
c.CmdClause.Flag("name", "The name of the Kinesis logging object").Short('n').Required().StringVar(&c.Input.Name)
return &c
}

// Exec invokes the application logic for the command.
func (c *DescribeCommand) Exec(in io.Reader, out io.Writer) error {
serviceID, source := c.manifest.ServiceID()
if source == manifest.SourceUndefined {
return errors.ErrNoServiceID
}
c.Input.ServiceID = serviceID

kinesis, err := c.Globals.Client.GetKinesis(&c.Input)
if err != nil {
return err
}

fmt.Fprintf(out, "Service ID: %s\n", kinesis.ServiceID)
fmt.Fprintf(out, "Version: %d\n", kinesis.ServiceVersion)
fmt.Fprintf(out, "Name: %s\n", kinesis.Name)
fmt.Fprintf(out, "Stream name: %s\n", kinesis.StreamName)
fmt.Fprintf(out, "Region: %s\n", kinesis.Region)
fmt.Fprintf(out, "Access key: %s\n", kinesis.AccessKey)
fmt.Fprintf(out, "Secret key: %s\n", kinesis.SecretKey)
fmt.Fprintf(out, "Format: %s\n", kinesis.Format)
fmt.Fprintf(out, "Format version: %d\n", kinesis.FormatVersion)
fmt.Fprintf(out, "Response condition: %s\n", kinesis.ResponseCondition)
fmt.Fprintf(out, "Placement: %s\n", kinesis.Placement)
Comment on lines +46 to +56
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should create a new /pkg/text/lines.go to make it easier to render output for newline information (like logging endpoints) and then we'd also be consistent with other packages in the CLI that use the text package.

Something generic like this would suffice: https://play.golang.org/p/csgKDzLZPeu

Which means you could do something like:

import "github.com/fastly/cli/pkg/text"

text.PrintLines(out, text.Lines{
    "Service ID": kinesis.ServiceID,
    "Version": kinesis.ServiceVersion,
    "Name": kinesis.Name,
    ...
})

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@phamann what do you think about this...

If we combined this generic implementation (see my comment above) with textio.NewPrefixWriter then we could probably end up reducing quite a few of the existing Print<T> functions (e.g. PrintBackend, PrintDictionaryItemKV etc).

You could even take it a step further and add in the ability to pass in an optional callback function for those edge cases such as PrintDictionaryItem where they only print lines that have values (although to be fair that's probably logic you'd shift back up into the consuming package). But I'm just 'brain dumping' ideas at the moment.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 to this idea from me. I'd say this is probably out of scope for this PR, but I think it would be a good change to implement broadly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree on both accounts:

  1. Very useful abstraction and would reduce repetition across the codebase.
  2. Out of scope of this PR.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏻 👌🏻


return nil
}
3 changes: 3 additions & 0 deletions pkg/logging/kinesis/doc.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
// Package kinesis contains commands to inspect and manipulate Fastly service Kinesis
// logging endpoints.
package kinesis