Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding documentation for all the properties available in Genie #446

Merged
merged 1 commit into from
Jan 4, 2017
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
469 changes: 469 additions & 0 deletions genie-docs/src/docs/asciidoc/_properties.adoc

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Many people ask how Genie is deployed at Netflix on AWS. This section tries to e
used and how Genie integrates into the environment. Below is a diagram of how deployment looks at Netflix.

.Genie Netflix Deployment
image::deployment.png[Netflix Deployment, link="{imagesdir}deployment.png"]
image::deployment.png[Netflix Deployment, link="{imagesdir}/deployment.png"]

==== Components

Expand Down
20 changes: 10 additions & 10 deletions genie-docs/src/docs/asciidoc/concepts/_netflixExample.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,13 @@ downloaded into the job working directory at runtime.
"genie.id:bdp_h2prod_20161217_205111",
"genie.name:h2prod",
"sched:sla",
"ver:2.7.0",
"ver:2.7.2",
"type:yarn",
"misc:h2bonus3",
"misc:h2bonus2",
"misc:h2bonus1"
],
"version": "2.7.0",
"version": "2.7.2",
"user": "dataeng",
"name": "h2prod",
"description": null,
Expand Down Expand Up @@ -88,13 +88,13 @@ downloaded into the job working directory at runtime.
"tags": [
"sched:adhoc",
"misc:profiled",
"ver:2.4.0",
"ver:2.7.2",
"sched:sting",
"type:yarn",
"genie.name:h2query",
"genie.id:bdp_h2query_20161108_204556"
],
"version": "2.7.0",
"version": "2.7.2",
"user": "dataeng",
"name": "h2query",
"description": null,
Expand Down Expand Up @@ -852,7 +852,7 @@ selected using the `commandCriteria`
<6> Here you can see that they add the two files referenced in the `commandArgs` as dependencies. These files will be
downloaded in the root job directory parallel to the run script so they are accessible.

===== The Job
===== *The Job*

In this case the job was accepted by Genie for processing. Below is the actual job object containing fields the user
might care about. Some are copied from the initial request (like tags) and some are added by Genie.
Expand Down Expand Up @@ -989,7 +989,7 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output

TIP: Click image for full size

image::output.png[Genie Output Directory, link="{imagesdir}output.png"]
image::output.png[Genie Output Directory, link="{imagesdir}/output.png"]

====== The Run Script

Expand Down Expand Up @@ -1127,15 +1127,15 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie

TIP: Click image for full size

image::genie-dir.png[Genie Directory, link="{imagesdir}genie-dir.png"]
image::genie-dir.png[Genie Directory, link="{imagesdir}/genie-dir.png"]

Genie system logs go into the logs directory.

TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/logs

TIP: Click image for full size

image::genie-logs.png[Genie Logs Directory, link="{imagesdir}genie-logs.png"]
image::genie-logs.png[Genie Logs Directory, link="{imagesdir}/genie-logs.png"]

Of interest in here is the env dump file. This is convenient for debugging jobs. You can see all the environment
variables that were available right before Genie executed the final command to run the job in the run script.
Expand Down Expand Up @@ -1211,13 +1211,13 @@ TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/applicati

TIP: Click image for full size

image::hadoop-application.png[Hadoop App Contents, link="{imagesdir}hadoop-application.png"]
image::hadoop-application.png[Hadoop App Contents, link="{imagesdir}/hadoop-application.png"]

TIP: https://genieHost/output/SP.CS.FCT_TICKET_0054500815/output/genie/applications/spark161/dependencies/spark-1.6.1

TIP: Click image for full size

image::spark-application.png[Hadoop App Contents, link="{imagesdir}spark-application.png"]
image::spark-application.png[Hadoop App Contents, link="{imagesdir}/spark-application.png"]

==== Wrap Up

Expand Down
Binary file removed genie-docs/src/docs/asciidoc/images/deployment.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/genie-dir.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/genie-logs.png
Binary file not shown.
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/output.png
Binary file not shown.
Binary file removed genie-docs/src/docs/asciidoc/images/security.png
Binary file not shown.
Binary file not shown.
7 changes: 5 additions & 2 deletions genie-docs/src/docs/asciidoc/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,18 @@ v{revnumber}, {localdate}
:description: Reference documentation for Netflix OSS Genie
:keywords: genie, netflix, documentation, big data, cloud, oss, open source software
:toc: left
:toclevels: 2
:toclevels: 3
:doctype: book
:sectanchors:
:sectlinks:
:sectnums:
:sectnumlevels: 5
:linkattrs:
:icons: font
:stylesheet: rubygems.css
:stylesdir: stylesheets
:source-highlighter: highlight.js
:imagesdir: images/
:imagesdir: https://netflix.github.io/genie/images/3.0.0

== Introduction

Expand All @@ -30,3 +31,5 @@ https://netflix.github.io/genie/docs/{revnumber}/demo[Demo Guide].
WARNING: Work In Progress

include::concepts/_concepts.adoc[]

include::_properties.adoc[]
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ public String hostName() throws UnknownHostException {
@Bean(name = "genieRestTemplate")
public RestTemplate restTemplate(
@Value("${genie.http.connect.timeout:2000}") final int httpConnectTimeout,
@Value("${genie.http.connect.timeout:10000}") final int httpReadTimeout
@Value("${genie.http.read.timeout:10000}") final int httpReadTimeout
) {
final HttpComponentsClientHttpRequestFactory factory = new HttpComponentsClientHttpRequestFactory();
factory.setConnectTimeout(httpConnectTimeout);
Expand All @@ -112,8 +112,8 @@ public RestTemplate restTemplate(
* Get RetryTemplate.
*
* @param noOfRetries number of retries
* @param initialInterval initial interval for the backoff policy
* @param maxInterval maximum interval for the backoff policy
* @param initialInterval initial interval for the back-off policy
* @param maxInterval maximum interval for the back-off policy
* @return The retry template to use
*/
@Bean(name = "genieRetryTemplate")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,6 @@
import org.springframework.web.client.RestTemplate;

import java.io.IOException;
import java.nio.charset.Charset;

/**
* Beans and configuration specifically for MVC on AWS.
Expand All @@ -38,8 +37,6 @@
@Slf4j
public class AwsMvcConfig {

private static final Charset UTF_8 = Charset.forName("UTF-8");

// See: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AESDG-chapter-instancedata.html
protected String publicHostNameGet = "http://169.254.169.254/latest/meta-data/public-hostname";
protected String localIPV4HostNameGet = "http://169.254.169.254/latest/meta-data/local-ipv4";
Expand All @@ -54,7 +51,7 @@ public class AwsMvcConfig {
*/
@Bean
public String hostName(@Qualifier("genieRestTemplate") final RestTemplate restTemplate) throws IOException {
String result = null;
String result;
try {
result = restTemplate.getForObject(publicHostNameGet, String.class);
log.debug("AWS Public Hostname: {}", result);
Expand Down