SURF

Free Text Search across your infrastructure platforms via CLI.

It's like ... | grep but for your entire infrastructure!

S.U.R.F is an acronym for: Search-Unified-Recursive-Fast

Supported Platforms

• AWS Route53 and Targets
• AWS CloudControl
• AWS ACM
• AWS S3
• AWS DynamoDB
• Hashicorp Vault
• Hashicorp Consul KV
• ElasticSearch / AWS OpenSearch
• Logz.io
• Kubernetes - TODO

Table of Contents

• SURF
• Supported Platforms
• Table of Contents
• Overview
• Usage Examples
  • AWS Route53 Usage
  • AWS Cloud Control Usage
  • AWS ACM Usage
  • AWS DynamoDB Usage
  • AWS S3 Usage
  • Hashicorp Vault Usage
  • Hashicorp Consul Usage
  • ElasticSearch and OpenSearch Usage
  • Logz.io Usage
• Install
  • Brew
  • Download Binary
  • Install from Source
• Authentication
  • Supported Authentication Methods
• Version check
• How it Works
• Contributors

Overview

SURF is built for Infrastructure Engineers as a CLI tool that enables searching any pattern across different platforms. Usually, the results are returned with a direct web URL.

The search process depends on the context, for example: if you're searching in Vault it'll pattern match against keys. Instead, if you're searching in Route53 AWS a DNS address it'll return links to the targets behind it (e.g Load balancer).

Usage Examples

AWS Route53 Usage

Based on AWS Route53: Search what's behind domain api.my-corp.com:

surf r53 -q api.my-corp.com

AWS Cloud Control Usage

Cloud Control API allows searching across AWS resources in a resource agnostic manner. All resource types has CRUD and based on REST so we can list, get, create, delete.

The Supported resource types are auto generated into the code during every build see cloudformation_resources.gen.go.

List Supported resource types

Note: certain resources require additional fields to be specified, such as AWS::EKS::Addon And list-types will show the required fields (e.g ClusterName).

surf aws list-types

List Resources

Example: List all EKS clusters:

surf aws list --type eks::cluster

Search existing resources

surf aws search  -q <query>  -t [<resource-type>] -a [<field-key>=<field-val>]

Example search: resources containing the my-app anywhere and are of types eks or vpc:

surf aws search -q my-app -t eks -t vpc -a `ClusterName=MyClusterName`

Example Describe EKS cluster my-cluster:

surf aws get --type eks::cluster --id my-cluster

AWS ACM Usage

Search inside ACM Certificates in AWS.

Example search: containing a domain:

surf acm -q my-domain.com

Example search: certificate attached to a loab balancer:

surf acm -q 's:elasticloadbalancing:us-west-2:123:loadbalancer/app/alb' --filter-used-by

AWS DynamoDB Usage

Search free text data in DynamoDB

Supported Formats: protobuf, base64, json, binary, bytes.

Example: list existing tables

surf ddb --list-tables

Example: use -p for aws profile, -r for region

surf ddb -q val -t table -p my-aws-profile -r us-east-1

Example: search all tables with production in their name, where the data containing the pattern val

surf ddb -q val --all-tables -t production

Example: search all tables data containing the word val, output as JSON

surf ddb -q val --all-tables -o json

Example: stop on first match, search all tables data containing the word val

surf ddb -q val -t my-prefix-table --stop-first-match

AWS S3 Usage

Search inside S3 Buckets and Keys in AWS.

Example: Find all keys containing logs in all buckets containing the name prod-bucket:

surf s3  -q 'logs' -b 'prod-bucket'

Example: find all keys containng house with prefix key my

surf s3 --prefix my -q house -b my-bucket

Example: find all keys ending with .json and use non-default AWS profile:

surf s3 -q '\.json$' -b bucket-prefix -p my-aws-profile 

Optional: Configure a default bucket name (same as --bucket flag) to start search from (any regex pattern):

export SURF_S3_DEFAULT_MOUNT=<my-bucket-pattern>

Hashicorp Vault Usage

Search the query aws in Vault:

surf vault -q aws 

Configure a default mount to start search from in Vault:

export SURF_VAULT_DEFAULT_MOUNT=<my-default-mount>

Store LDAP auth on your OS keychain:

surf config

Hashicorp Consul Usage

Search all keys containing the substring server

surf consul -q "server"

Search under the scripts path for keys ending with .sh

surf consul --prefix scripts --query "\.sh$"

ElasticSearch and OpenSearch Usage

Search free text and/or KQL.

Supports Standard Elasticsearch and AWS Opensearch.

• Example: Search docs containing the term api return limit 40 results.

  surf es -q 'api' -l 40

• Example: Search in indexes prod-* and api-*

  surf es -q 'api' -i 'prod-*d' -i 'api-*'

• Example: Search docs containing the term api with client field and xyz* pattern and NOT containing the term staging

  surf es -q 'api AND client:xyz*' --nq staging

• Example: List Indexes

  surf es --list-indexes

• Example: Configure Elasticsearch default settings (use surf es --help for more):

  SURF_ELASTICSEARCH_URL
  SURF_ELASTICSEARCH_USERNAME='token-or-username-and-password'
  SURF_ELASTICSEARCH_PASSWORD='token-or-username-and-password'
  SURF_ELASTICSEARCH_TOKEN='token-or-username-and-password'
  SURF_ELASTICSEARCH_INDEXES='prod-*,api-*'

• Example: Store Username/Password or Token auth on your OS keychain:

  surf config

Logz.io Usage

Free text and KQL Search in logz.io, use the api website to get the address.

• Example: Search docs containing the word 'exception' with limit size 200

  surf logz -q 'exception' -l 200

• Example: Search docs containing the word 'something' across ALL sub-accounts matching production/automation

  surf logz -q 'something' --acc production --acc automation 

• Example: Search docs containing the word 'production', field errorCode with value Access* and are not containing 'dummy'

  surf logz -q 'production AND errorCode:Access*' --nq 'dummy'

• Example: Search docs across 10 day window with 2 days offset (e.g all matches between 12 days ago until 2 days ago)

  surf logz -o 2 -w 10d -q 'some pattern'

• Example: Configure Logz.io default settings (use surf logz --help for more):

  SURF_LOGZ_IO_TOKEN
  SURF_LOGZ_IO_URL
  SURF_LOGZ_IO_ACCOUNT_IDS

• Example: Store Username/Password or Token auth on your OS keychain:

  surf config

Install

Brew

MacOS (and ubuntu supported) installation via Brew:

brew tap isan-rivkin/toolbox
brew install surf

Download Binary

1. from releases

2. Move the binary to global dir and change name to surf:

cd <downloaded zip dir>
mv surf /usr/local/bin

Install from Source

git clone git@github.com:Isan-Rivkin/surf.git
cd surf
go run main.go

Authentication

Please open a PR and request additional methods if you need.

Supported Authentication Methods

• Vault - LDAP (run $surf config )
• AWS - via profile on ~/.aws/credentials file
• Consul - None
• Elasticsearch / Opensearch - User/Pass or Token (run $surf config or surf es --help)
• Logz.io - Token (run $surf config or surf logz --help)

Version check

The CLI will query github.com to check if there is a newer version and print out a message to the terminal.

If you wish to opt out set the environment variable SURF_VERSION_CHECK=false.

No Data is collected it is purely github.com query.

How it Works

Contributors

• @AliRamberg
• @Isan-Rivkin