Tony History Server #62

pdtran3k6 · 2018-10-30T23:33:52Z

Added custom script for starting THS.
Refactored + Modularized code.

… code;

…n3k6/TonY into phattran_tonyHistoryServer

oliverhu · 2018-10-31T04:43:44Z

tony-history-server/app/controllers/JobsConfigPageController.java

+  public Result index(String jobId) {
+    List<JobConfig> listOfConfigs = new ArrayList<>();
+    FileSystem myFs = getFs();
+    String tonyHistoryFolder = config.getString("tony.historyFolder");


make tony.historyFolder constant

Something like this: https://github.com/linkedin/TonY/blob/master/tony-core/src/main/java/com/linkedin/tony/Constants.java?

Hmm so in that case can I put Constants in utils?

Actually, on a second thought, I don't think we should put it in Constants file, as tony.historyFolder is different for dev/prod mode.

tony.historyFolder is the name of the parameter, which is constant. The actual value (which will be different in dev/prod) comes from the application.conf file (or later from $TONY_CONF_DIR/tony-site.xml).

Thus, I think tony.historyFolder can still be in the Constants class.

tony-history-server/app/controllers/JobsConfigPageController.java

oliverhu · 2018-10-31T04:45:48Z

tony-history-server/app/utils/HdfsUtils.java

+/**
+ * The class handles all HDFS file operations
+ */
+public class HdfsUtils {


this seems generic for all modules. We might need a component for all utils. Something like hadoop-common.

oliverhu · 2018-10-31T04:46:07Z

tony-history-server/app/utils/HdfsUtils.java

+    return localFileName;
+  }
+
+  static List<Path> getValidPaths(FileStatus[] lsJobDir, Predicate<FileStatus> fn) {


We need tests for methods in this class

I'll look into how tests are written for a Play application and I'll add some soon.

These utility methods should be easy to test since you can just instantiate the input objects, directly call the methods, and then assert things about the returned output.

tony-history-server/app/utils/SecurityUtils.java

erwa · 2018-11-07T01:49:00Z

I think the two main things I'd like to see before we commit this are:

tests for the Utils classes
Javadoc for the Utils methods

pdtran3k6 · 2018-11-07T22:43:39Z

For dbc7615, I didn't write Javadoc for the last method, as me and @erwa discussed offline that it would be better to refactor the method to only be reading a specific file type (config file, which is always xml). Will update Javadoc after refactoring.

erwa · 2018-11-07T23:37:19Z

tony-history-server/app/utils/HdfsUtils.java

@@ -24,6 +25,12 @@
 public class HdfsUtils {
  private static final Logger.ALogger LOG = Logger.of(HdfsUtils.class);

+  /**
+   * Check to see if a HDFS path is valid.


Please define what "valid" means. It looks like this method just tests for path existence, in which case, why do we need this method? Why not just call fs.exists(filePath) directly?

You're correct, but imo for the sake of code modularity, we should have the method. Calling fs.exists directly would require the try/catch block every time you call it, which makes the caller method more verbose than it should be.

Ah, I see. Avoiding try/catch blocks is a reasonable reason for adding this method.

Can we still update the description and replace is valid with exists? I think that's clearer. Also, I suggest renaming the method pathExists.

erwa · 2018-11-07T23:38:00Z

tony-history-server/app/utils/HdfsUtils.java

@@ -33,6 +40,12 @@ static boolean isPathValid(FileSystem fs, Path filePath) {
    }
  }

+  /**
+   * Retrieve the content of the file on HDFS.


I think you should mention as a string in the description.

Noted. Fixed: 610a368

erwa · 2018-11-07T23:41:22Z

tony-history-server/app/utils/HdfsUtils.java

@@ -48,13 +61,26 @@ static String contentOfHdfsFile(FileSystem fs, Path filePath) {
    }
  }

+  /**
+   * Return all paths that satisfy the <code>fn</code> predicate.


It doesn't make a difference here, but in general, {@code ...} is preferable to using <code>...</code> as {@code ...} lets you embed < and > without escaping them as < and >. Ref: https://stackoverflow.com/a/1671106/1128392

I was considering using backticks (`) to wrap the variable name in the doc, but I think I saw it somewhere where they use <code></code> as the variable's wrapper. Which one do you guys prefer? @erwa @oliverhu @hungj

Backticks is non-standard and won't cause the enclosed text to get rendered as monospace when Javadoc HTML is generated. Let's stick with {@code ...}.

Resolved: 63c80af

tony-history-server/test/utils/TestHdfsUtils.java

tony-history-server/app/utils/HdfsUtils.java

tony-history-server/test/utils/TestHdfsUtils.java

erwa · 2018-11-08T03:06:29Z

For dbc7615, I didn't write Javadoc for the last method, as me and @erwa discussed offline that it would be better to refactor the method to only be reading a specific file type (config file, which is always xml). Will update Javadoc after refactoring.

I think we don't need the method at all. We're designing the file naming conventions and folder layout, so we know what they are and can directly check for and read the expected files. (Also see my comment on HdfsUtils.java line 84.)

…d Security interface

oliverhu · 2018-11-09T17:58:50Z

There has been 59 commits in this PR.. shall we clean up the existing and move forward with new features?

pdtran3k6 · 2018-11-09T18:00:24Z

@oliverhu 💯yeah I think so too... I'll make a new branch after merging this PR.

erwa · 2018-11-09T20:21:33Z

Yeah, I plan to review and hopefully commit this this afternoon.

erwa

Looks good. A few mostly minor comments that can be addressed in follow-up PRs. Merging this now.

tony-history-server/app/controllers/JobsMetadataPageController.java

tony-history-server/app/hadoop/Security.java

tony-history-server/test/utils/TestHdfsUtils.java

erwa · 2018-11-09T23:16:43Z

Please do all new enhancements, feature work, and bug fixes in separate PRs (use a separate branch for each). Try to keep each PR focused on a single thing -- makes it easier to review and merge.

pdtran3k6 and others added 22 commits October 4, 2018 18:08

Skeleton for Tony History Server

a16901a

Updated logo

d3df0b6

Removed old code

db0ebaf

Added build scripts, docs, assets, config files

8bddd93

Added main app + basic tests

118cd6c

Modified .gitignore to exclude files in tony-history-server

5c3a412

Updated .gitignore to ignore deploy.log

16cc0f3

Updated docs + deploy script

2f7648f

Removed extra gradle bin + Updated build.gradle at root and ths

33b6e69

Removed clutter in routes config

b73333e

Removed old Qihoo's files

a01e56e

Fixed indentation

79992b2

Fixed indentation pt.2

6e9fa7b

Removed gradlew.bat

27a8402

Updated indentation; Reverted back to original route name; Refactored…

555eb41

… code;

Modified Scala version

5572106

Cleaned up piping syntax for deploy script

500e580

Reverted script piping syntax; Modified indentation

b70834c

Merge branch 'master' into phattran_tonyHistoryServer

288104b

Added server starting script for distribution mode

4ae7d5b

Refactored code

76c241f

Merge branch 'phattran_tonyHistoryServer' of https://github.com/pdtra…

a60fbdf

…n3k6/TonY into phattran_tonyHistoryServer

pdtran3k6 requested review from erwa, oliverhu and hungj October 30, 2018 23:33

pdtran3k6 self-assigned this Oct 30, 2018

Merge branch 'master' into phattran_tonyHistoryServer

1b2f721

pdtran3k6 changed the title ~~Phattran tony history server~~ Tony History Server Oct 30, 2018

oliverhu reviewed Oct 31, 2018

View reviewed changes

Modified gradle and deploy scripts to take in projects name dynamically

5a1cbc6

pdtran3k6 added 6 commits November 4, 2018 14:07

Reformatted CSS; Updated logo + banner photo; Updated color

3f419ac

Removed default svg

17776c5

Renamed config controller

d2b7f7d

Modified catch blocks to catch more specific exceptions

f84caff

Refactored security module

89bd2fd

Ignore xml files generated by Tony AM

e27322b

Added Javadoc for most methods

dbc7615

erwa reviewed Nov 7, 2018

View reviewed changes

pdtran3k6 added 2 commits November 7, 2018 16:00

Added some regression tests for HdfsUtils

ef99877

Removed unnecessary exception

610a368

erwa reviewed Nov 8, 2018

View reviewed changes

tony-history-server/test/utils/TestHdfsUtils.java Outdated Show resolved Hide resolved

tony-history-server/app/utils/HdfsUtils.java Outdated Show resolved Hide resolved

tony-history-server/test/utils/TestHdfsUtils.java Show resolved Hide resolved

pdtran3k6 added 6 commits November 8, 2018 16:46

Refactored HdfsUtils; Updated HdfsUtils tests

5080c61

Ignore test build folders

83329ac

Uploaded test environment

ec6e1d7

Fixed .gitignore; Updated css

c488415

Split Security module into submodules to decrease coupling; Refactore…

565fdc1

…d Security interface

Updated docs

63c80af

erwa approved these changes Nov 9, 2018

View reviewed changes

Merge branch 'master' into phattran_tonyHistoryServer

d59e0fe

erwa merged commit 209f1f3 into tony-framework:master Nov 9, 2018

erwa mentioned this pull request Nov 9, 2018

TonY History Server should use in-memory cache to avoid pounding the NameNode every time the jobs list page is loaded #72

Closed

pdtran3k6 deleted the phattran_tonyHistoryServer branch November 11, 2018 00:56

pdtran3k6 mentioned this pull request Nov 11, 2018

Fix minor stuff in THS code + tests. #73

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tony History Server #62

Tony History Server #62

pdtran3k6 commented Oct 30, 2018

oliverhu Oct 31, 2018

pdtran3k6 Oct 31, 2018

oliverhu Oct 31, 2018

pdtran3k6 Oct 31, 2018

pdtran3k6 Nov 1, 2018

erwa Nov 1, 2018 •

edited

Loading

oliverhu Oct 31, 2018

oliverhu Oct 31, 2018

pdtran3k6 Oct 31, 2018

erwa Oct 31, 2018

erwa commented Nov 7, 2018

pdtran3k6 commented Nov 7, 2018

erwa Nov 7, 2018

pdtran3k6 Nov 8, 2018

erwa Nov 8, 2018

erwa Nov 7, 2018

pdtran3k6 Nov 8, 2018

erwa Nov 7, 2018

pdtran3k6 Nov 8, 2018

erwa Nov 8, 2018

pdtran3k6 Nov 9, 2018

erwa commented Nov 8, 2018

oliverhu commented Nov 9, 2018

pdtran3k6 commented Nov 9, 2018

erwa commented Nov 9, 2018

erwa left a comment

erwa commented Nov 9, 2018

Tony History Server #62

Tony History Server #62

Conversation

pdtran3k6 commented Oct 30, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erwa Nov 1, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erwa commented Nov 7, 2018

pdtran3k6 commented Nov 7, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erwa commented Nov 8, 2018

oliverhu commented Nov 9, 2018

pdtran3k6 commented Nov 9, 2018

erwa commented Nov 9, 2018

erwa left a comment

Choose a reason for hiding this comment

erwa commented Nov 9, 2018

erwa Nov 1, 2018 •

edited

Loading