Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Quick Start Guide

copperlight edited this page · 3 revisions

Table of Contents

Configure AWS Access

You must configure credentials so that Edda can crawl AWS resources. The best method for obtaining temporary keys for local use is to create an EC2 instance with an IAM role and pull a copy of the rotating keys from the instance metadata. Once you have the keys, you can set them as environment variables:

 $ export AWS_ACCESS_KEY_ID=yourAccessKey
 $ export AWS_SECRET_KEY=yourSecretKey

Or you can add them to the edda.properties file located at either $EDDA_HOME/src/main/resources/edda.properties or $TOMCAT_HOME/webapps/$EDDA_CONTEXT/WEB-INF/classes:

edda.aws.accessKey=yourAccessKey
edda.aws.secretKey=yourSecretKey

You can create AWS accounts, configure roles and allocate keys with the Identity and Access Management (IAM) service.

Build Edda with Gradle

If you are using Java 7, then the default PermGen size is too small to compile Scala with Gradle and you need to increase it to 256M. If you are using Java 8, then you do not need to configure additional PermGen space because this configuration is no longer supported on the newer JVM.

 $ export JAVA_OPTS="-Xmx1g -XX:MaxPermSize=256M"

Clone the repository from GitHub:

 $ git clone https://github.com/Netflix/edda.git
 Cloning into 'edda'...
 remote: Counting objects: 1740, done.
 remote: Compressing objects: 100% (613/613), done.
 remote: Total 1740 (delta 772), reused 1619 (delta 651)
 Receiving objects: 100% (1740/1740), 358.40 KiB | 274 KiB/s, done.
 Resolving deltas: 100% (772/772), done.

Build Edda:

 $ ./gradlew build
 Download https://github.com/Netflix-Skunkworks/build-repo/raw/master/releases/gradle-release/gradle-release/1.0-SNAPSHOT/gradle-release-1.0-SNAPSHOT.jar
 :compileJava UP-TO-DATE
 :compileScala UP-TO-DATE
 :processResources UP-TO-DATE
 :classes UP-TO-DATE
 :jar UP-TO-DATE
 :javadoc UP-TO-DATE
 :javadocJar UP-TO-DATE
 :scaladoc UP-TO-DATE
 :scaladocJar UP-TO-DATE
 :sourcesJar UP-TO-DATE
 :war UP-TO-DATE
 :signArchives SKIPPED
 :assemble UP-TO-DATE
 :licenseMain UP-TO-DATE
 :licenseTest UP-TO-DATE
 :compileTestJava UP-TO-DATE
 :compileTestScala
 :processTestResources
 :testClasses
 :test
 :check
 :build
 
 BUILD SUCCESSFUL
 
 Total time: 58.437 secs

Download and Start MongoDB

MongoDB is used for leader election in Edda clusters and storing a persistent copy of the crawl results. Download a MongoDB distribution. The following examples will demonstrate installing and running MongoDB on a Mac OSX system, however the instructions should hold for other Unix systems. Windows will be different.

 $ curl -O http://fastdl.mongodb.org/osx/mongodb-osx-x86_64-2.2.1.tgz
 $ tar xzf mongodb.tgz
 $ mkdir ./mongo-data
 $ nohup mongodb-*/bin/mongod --dbpath=./mongo-data >mongod.log 2>&1 &

Configure Edda

The defaults should work for most simple cases. If your primary AWS region is not us-west-1 then you will need to change the edda.regions properties in the configuration file:

 $ vi ./src/main/resources/edda.properties

See the Configuration page for details on the various configuration options.

Run Edda with Gradle

 $ ./gradlew jettyRun

You should see Edda Start up and start processing all the resources under your AWS account for the region you specified. The output will be pretty verbose, it will log for each new record it sees, each record that is no longer active, and it will log unified diff output for all changes that are detected. The first time Edda is run it can take several minutes to insert all of the AWS resources into MongoDB, depending on how many resources you have.

Test the Edda API

In a different shell, you can start playing with the REST APIs on your running Edda server. Gradle starts a Jetty server on http://localhost:8080/edda for the application.

Find out how many instances you have:

 $ export EDDA=http://localhost:8080/edda/api/v2
 
 $ curl $EDDA/view/instances
 ["i-0123456789","i-012345678a","i-012345678b",...]

Make the output easier to read with the pretty-printer:

 $ curl "$EDDA/view/instances;_pp"
 [
   "i-0123456789",
   "i-012345678a",
   "i-012345678b",
   ...
 ]

Find an instance with a known publicIpAddress:

 $ curl "$EDDA/view/instances;publicIpAddress=1.2.3.4"
 ["i-012345678a"]

See the details of that instance:

 $ curl "$EDDA/view/instances/i-012345678a;_pp"
 {
   "amiLaunchIndex": 0,
   "architecture": "x86_64",
   "blockDeviceMappings": [],
   "clientToken": "",
   "imageId": "ami-0123456789",
   "instanceId": "i-012345678a",
   "instanceLifecycle": null,
   "instanceType": "m1.xlarge",
   "kernelId": "aki-0123456789",
   "keyName": "keypair",
   "launchTime": 1276882740000,
   "license": null,
   "monitoring": {
     "state": "enabled"
   },
   "placement": {
     "availabilityZone": "us-east-1a",
     "groupName": "",
     "tenancy": "default"
   },
   "platform": null,
   "privateDnsName": "domU-AA-BB-CC-DD-EE-FF.compute-1.internal",
   "privateIpAddress": "10.10.10.1",
   "productCodes": [],
   "publicDnsName": "ec2-1-2-3-4.compute-1.amazonaws.com",
   "publicIpAddress": "1.2.3.4",
   "ramdiskId": "ari-0123456789",
   "rootDeviceName": null,
   "rootDeviceType": "instance-store",
   "securityGroups": [{
     "groupId": "sg-0123456789",
     "groupName": "corp"
   }],
   "sourceDestCheck": null,
   "spotInstanceRequestId": null,
   "state": {
     "code": 16,
     "name": "running"
   },
   "stateReason": null,
   "stateTransitionReason": "",
   "subnetId": null,
   "tags": [],
   "virtualizationType": "paravirtual",
   "vpcId": null
 }

Use Field Selectors to pull out the instance state and privateIpAddress:

 $ curl "$EDDA/view/instances/i-012345678a;_pp:(state:(name),privateIpAddress)"
 {
    "privateIpAddress": "10.10.10.1",
    "state": {
      "name": "running"
    }
 }

Further Reading

  • Continue experimenting with the REST APIs and matrix arguments.
  • Configuration options are available to customize the Edda service.
Something went wrong with that request. Please try again.