Skip to content

Commit

Permalink
document v2 API (#236)
Browse files Browse the repository at this point in the history 
* document v2 API

* Many minor updates, and added Dependencies

* fix travis lint warnings
  • Loading branch information
@gfr10598
gfr10598 committed May 2, 2019
1 parent 373997e commit af5e194
Show file tree
Hide file tree
Showing 2 changed files with 91 additions and 49 deletions.
36 changes: 18 additions & 18 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -34,16 +34,16 @@ before_install:
# This installs gcloud, sets up devappserver.
- $TRAVIS_BUILD_DIR/annotator.sh

# NB: Encrypted values are not defined in forks or pull requests.
#
# Decrypt the tar archive containing the GCP service account key files.
# After unpacking, there should be one service account key file for every GCP
# project referenced in the "deploy" section. These keys authenticate the
# gcloud deploy operations.
#
# Reusing the etl-travis-deploy service accounts, since we need identical
# capabilities. The tar file contains newly created keys for each of the three
# service accounts.
# NB: Encrypted values are not defined in forks or pull requests.
#
# Decrypt the tar archive containing the GCP service account key files.
# After unpacking, there should be one service account key file for every GCP
# project referenced in the "deploy" section. These keys authenticate the
# gcloud deploy operations.
#
# Reusing the etl-travis-deploy service accounts, since we need identical
# capabilities. The tar file contains newly created keys for each of the three
# service accounts.
- travis/decrypt.sh "$encrypted_ca29bb0d9214_key" "$encrypted_ca29bb0d9214_iv"
keys/service-accounts.tar.enc /tmp/service-accounts.tar /tmp
- if [[ -r /tmp/service-accounts.tar ]] ; then
Expand All @@ -64,7 +64,7 @@ cache:
- $HOME/google-cloud-sdk/

script:
# This sets up the dev appserver for the aetest support used by unit tests.
# This sets up the dev appserver for the aetest support used by unit tests.
- if [[ -r "${HOME}/gcloud-service-key.json" ]]; then
export GOOGLE_APPLICATION_CREDENTIALS="${HOME}/gcloud-service-key.json";
fi
Expand All @@ -84,11 +84,11 @@ script:
# Deploy steps never trigger on a new Pull Request. Deploy steps will trigger
# after a merge with matching "on:" conditions.
deploy:
######################################################################
## Service: annotation-service -- AppEngine Flexible Environment.
######################################################################
## Service: annotation-service -- AppEngine Flexible Environment.

# SANDBOX: Allows any branch with sandbox-*, to trigger deploying that
# branch to sandbox for pre-review testing.
# SANDBOX: Allows any branch with sandbox-*, to trigger deploying that
# branch to sandbox for pre-review testing.
- provider: script
script:
$TRAVIS_BUILD_DIR/travis/deploy_app_legacy_keyfile.sh mlab-sandbox /tmp/mlab-sandbox.json $TRAVIS_BUILD_DIR annotator.yaml
Expand All @@ -99,7 +99,7 @@ deploy:
condition: "$TRAVIS_EVENT_TYPE == push"


# STAGING: Should trigger AFTER code review and commit to master branch.
# STAGING: Should trigger AFTER code review and commit to master branch.
- provider: script
script:
$TRAVIS_BUILD_DIR/travis/deploy_app_legacy_keyfile.sh mlab-staging /tmp/mlab-staging.json $TRAVIS_BUILD_DIR annotator.yaml
Expand All @@ -108,8 +108,8 @@ deploy:
repo: m-lab/annotation-service
branch: master

# PROD: Should trigger AFTER code review and commit to master branch. Triggers
# when tagged with prod-*.
# PROD: Should trigger AFTER code review and commit to master branch. Triggers
# when tagged with prod-*.
- provider: script
script:
$TRAVIS_BUILD_DIR/travis/deploy_app_legacy_keyfile.sh mlab-oti /tmp/mlab-oti.json $TRAVIS_BUILD_DIR annotator.yaml
Expand Down
104 changes: 73 additions & 31 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,18 +4,84 @@

Annotation integration service provides geolocation for IPv4 and IPv6 MaxMind databases from Google Cloud Storage.

If an annotation request is dated prior to August 2017, location data will be derived from
MaxMind GeoLiteLatest databases. Otherwise, data will be provided by
MaxMind GeoLite2 databases. The discrepencies between provided databases are
## API

### v1 - Deprecated

This is the original, deprecated API, which includes an unwrapped array of
RequestData objects. Its use is discouraged. Please use the v2 API, which
has better support for batch requests, including returning the date of the
Annotator used to provide the annotations.

### v2

The v2 api introduces a standard wrapper struct, beginning with a string
specifying the version identifier, and an Info string that may include
arbitrary request info, e.g. for use in tracing or debugging.
It is described in the api/v2 package in the api/v2 directory. The recommended
GetAnnotations function is only available in the v2 package.

### Response contents

Annotatation service will respond with the following data:
- IP Address range
- Postal Code
- Latitude
- Longitude
- Continent Code
- Country Code
- Country Name
- Metro Code
- City Name

---

## Code structure

The code is divided into the following packages (organized in rough order of dependencies, except for api package):

- api - defines external API, including GetAnnotations() call which handles composing and sending requests, with retries.
- manager - handles caching of Annotators
- directory - used by manager to create and keep track of CompositeAnnotators.
- handler - receives incoming requests, handles marshalling, unmarshalling, interpretation of requests.
- geoloader - maintains directory of available MaxMind (GEO) and Routeview (ASN) files, and selects which file(s) to use for a given date. (Needs a lot of renaming)
- asn - handles details of interpreting RouteViews ASN files, and creating ASN annotators.
- geolite2v2 and legacy - handle details of interpreting MaxMind files and creating annotators.
Currently this is divided into two packages, but should be merged.
- loader - handles files downloads and decompression
- iputil - general IP utility functions that are used across asn, legacy geo,
and geolite2 datasets.
- metrics - all metric definitions.

### Dependencies (as of April 2019)

(higher depends on lower, left -> depends on right)

- main.go
- manager -> handler, directory
- geoloader -> asn, geolite2v2, legacy
- iputils -> loader
- api, metrics

---

## Maxmind Dataset details

If an annotation request is dated prior to August 2017, location data will be
derived from MaxMind GeoLiteLatest databases. Otherwise, data will be provided
by MaxMind GeoLite2 databases. The discrepencies between provided databases are
provided below.

MaxMind GeoLiteLatest databases include:

1. GeoLiteCity-Blocks.csv

- StartIPNum IPv4
- EndIPNum IPv4
- GeonameID

1. GeoLiteCity-Location.csv

- GeonameID
- Country Code
- City
Expand All @@ -27,13 +93,16 @@ MaxMind GeoLiteLatest databases include:
MaxMind GeoLite2 databases include:

1. GeoLite2-City-Blocks-IPv4.csv & GeoLite2-City-Blocks-IPv6.csv

- IP network (CIDR Format)
- GeonameID (identifies end user location)
- Registered Country Geoname ID (identifies country where IP address is
registered to an ISP)
- Latitude
- Longitude

1. GeoLite2-City-Locations-en.csv

- GeonameID
- Continent Code
- Country ISO
Expand All @@ -44,31 +113,14 @@ MaxMind GeoLite2 databases include:
GeonameID is the same with Registered Country Geoname ID most of time, but with some exceptions.
Either GeonameID or Registered Country Geoname ID could be not available for some IP addresses.

Important descrepencies to note include:
Important discrepencies to note include:

1. GeoLite2 databases provides a network IP range in CIDR format while
GeoLiteLatest databases provide an IP range in decimal format.
1. GeoLite2 provides both end user location as well as country registration
information while GeoLiteLatest includes only end user location.
www.maxmind.com/en/geoip2-precision-city-service

Annotatation service will respond with the following data:

- IP Address range
- Postal Code
- Latitude
- Longitude
- Continent Code
- Country Code
- Country Name
- Metro Code
- City Name

## The directory package

The directory package provides tools to create a list of CompositeAnnotator (CA) wrappers.
directory.GetAnnotator(date time.Time) returns an appropriate CA for the given date.

### CompositeAnnotator

CompositeAnnotator encapsulates multiple component api.Annotator objects, and satisfies the
Expand All @@ -95,13 +147,3 @@ type CachingLoader interface {
}
```

TODO: Need to implement the CachingLoaders.

### Generator
```
func NewGenerator(v4, v6, g2 *CachingLoader)
func (gen *Generator) Update() error // Reloads all lists
func (gen *Generator) Generate() []Annotator // constructs list of CompositeAnnotators
```

0 comments on commit af5e194

Please sign in to comment.