Skip to content

Commit

Permalink
Adding documentation for all the properties available in Genie (#446)
Browse files Browse the repository at this point in the history
  • Loading branch information
tgianos committed Jan 4, 2017
1 parent 3e14e7c commit 56032a2
Show file tree
Hide file tree
Showing 13 changed files with 489 additions and 20 deletions.
469 changes: 469 additions & 0 deletions genie-docs/src/docs/asciidoc/_properties.adoc

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Many people ask how Genie is deployed at Netflix on AWS. This section tries to e
used and how Genie integrates into the environment. Below is a diagram of how deployment looks at Netflix.

.Genie Netflix Deployment
image::deployment.png[Netflix Deployment, link="{imagesdir}deployment.png"]
image::deployment.png[Netflix Deployment, link="{imagesdir}/deployment.png"]

==== Components

Expand Down
20 changes: 10 additions & 10 deletions genie-docs/src/docs/asciidoc/concepts/_netflixExample.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,13 @@ downloaded into the job working directory at runtime.
"genie.id:bdp_h2prod_20161217_205111",
"genie.name:h2prod",
"sched:sla",
"ver:2.7.0",
"ver:2.7.2",
"type:yarn",
"misc:h2bonus3",
"misc:h2bonus2",
"misc:h2bonus1"
],
"version": "2.7.0",
"version": "2.7.2",
"user": "dataeng",
"name": "h2prod",
"description": null,
Expand Down Expand Up @@ -88,13 +88,13 @@ downloaded into the job working directory at runtime.
"tags": [
"sched:adhoc",
"misc:profiled",
"ver:2.4.0",
"ver:2.7.2",
"sched:sting",
"type:yarn",
"genie.name:h2query",
"genie.id:bdp_h2query_20161108_204556"
],
"version": "2.7.0",
"version": "2.7.2",
"user": "dataeng",
"name": "h2query",
"description": null,
Expand Down Expand Up @@ -852,7 +852,7 @@ selected using the `commandCriteria`
<6> Here you can see that they add the two files referenced in the `commandArgs` as dependencies. These files will be
downloaded in the root job directory parallel to the run script so they are accessible.

===== The Job
===== *The Job*

In this case the job was accepted by Genie for processing. Below is the actual job object containing fields the user
might care about. Some are copied from the initial request (like tags) and some are added by Genie.
Expand Down Expand Up @@ -989,7 +989,7 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output

TIP: Click image for full size

image::output.png[Genie Output Directory, link="{imagesdir}output.png"]
image::output.png[Genie Output Directory, link="{imagesdir}/output.png"]

====== The Run Script

Expand Down Expand Up @@ -1127,15 +1127,15 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie

TIP: Click image for full size

image::genie-dir.png[Genie Directory, link="{imagesdir}genie-dir.png"]
image::genie-dir.png[Genie Directory, link="{imagesdir}/genie-dir.png"]

Genie system logs go into the logs directory.

TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/logs

TIP: Click image for full size

image::genie-logs.png[Genie Logs Directory, link="{imagesdir}genie-logs.png"]
image::genie-logs.png[Genie Logs Directory, link="{imagesdir}/genie-logs.png"]

Of interest in here is the env dump file. This is convenient for debugging jobs. You can see all the environment
variables that were available right before Genie executed the final command to run the job in the run script.
Expand Down Expand Up @@ -1211,13 +1211,13 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/applicati

TIP: Click image for full size

image::hadoop-application.png[Hadoop App Contents, link="{imagesdir}hadoop-application.png"]
image::hadoop-application.png[Hadoop App Contents, link="{imagesdir}/hadoop-application.png"]

TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/applications/spark161/dependencies/spark-1.6.1

TIP: Click image for full size

image::spark-application.png[Hadoop App Contents, link="{imagesdir}spark-application.png"]
image::spark-application.png[Hadoop App Contents, link="{imagesdir}/spark-application.png"]

==== Wrap Up

Expand Down
Binary file removed genie-docs/src/docs/asciidoc/images/deployment.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/genie-dir.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/genie-logs.png
Binary file not shown.
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/output.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/security.png
Binary file not shown.
Binary file not shown.
7 changes: 5 additions & 2 deletions genie-docs/src/docs/asciidoc/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,18 @@ v{revnumber}, {localdate}
:description: Reference documentation for Netflix OSS Genie
:keywords: genie, netflix, documentation, big data, cloud, oss, open source software
:toc: left
:toclevels: 2
:toclevels: 3
:doctype: book
:sectanchors:
:sectlinks:
:sectnums:
:sectnumlevels: 5
:linkattrs:
:icons: font
:stylesheet: rubygems.css
:stylesdir: stylesheets
:source-highlighter: highlight.js
:imagesdir: images/
:imagesdir: https://netflix.github.io/genie/images/3.0.0

== Introduction

Expand All @@ -30,3 +31,5 @@ https://netflix.github.io/genie/docs/{revnumber}/demo[Demo Guide].
WARNING: Work In Progress

include::concepts/_concepts.adoc[]

include::_properties.adoc[]
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ public String hostName() throws UnknownHostException {
@Bean(name = "genieRestTemplate")
public RestTemplate restTemplate(
@Value("${genie.http.connect.timeout:2000}") final int httpConnectTimeout,
@Value("${genie.http.connect.timeout:10000}") final int httpReadTimeout
@Value("${genie.http.read.timeout:10000}") final int httpReadTimeout
) {
final HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setConnectTimeout(httpConnectTimeout);
Expand All @@ -112,8 +112,8 @@ public RestTemplate restTemplate(
* Get RetryTemplate.
*
* @param noOfRetries number of retries
* @param initialInterval initial interval for the backoff policy
* @param maxInterval maximum interval for the backoff policy
* @param initialInterval initial interval for the back-off policy
* @param maxInterval maximum interval for the back-off policy
* @return The retry template to use
*/
@Bean(name = "genieRetryTemplate")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
import org.springframework.web.client.RestTemplate;

import java.io.IOException;
import java.nio.charset.Charset;

/**
* Beans and configuration specifically for MVC on AWS.
Expand All @@ -38,8 +37,6 @@
@Slf4j
public class AwsMvcConfig {

private static final Charset UTF_8 = Charset.forName("UTF-8");

// See: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
protected String publicHostNameGet = "http://169.254.169.254/latest/meta-data/public-hostname";
protected String localIPV4HostNameGet = "http://169.254.169.254/latest/meta-data/local-ipv4";
Expand All @@ -54,7 +51,7 @@ public class AwsMvcConfig {
*/
@Bean
public String hostName(@Qualifier("genieRestTemplate") final RestTemplate restTemplate) throws IOException {
String result = null;
String result;
try {
result = restTemplate.getForObject(publicHostNameGet, String.class);
log.debug("AWS Public Hostname: {}", result);
Expand Down

0 comments on commit 56032a2

Please sign in to comment.