Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[SPARK-3454] separate json endpoints for data in the UI #5940

Closed
wants to merge 6 commits into from
Closed

[SPARK-3454] separate json endpoints for data in the UI #5940

wants to merge 6 commits into from

Conversation

squito
Copy link
Contributor

@squito squito commented May 6, 2015

Exposes data available in the UI as json over http. Key points:

  • new endpoints, handled independently of existing XyzPage classes. Root entrypoint is JsonRootResource
  • Uses jersey + jackson for routing & converting POJOs into json
  • tests against known results in HistoryServerSuite
  • also fixes some minor issues w/ the UI -- synchronizing on access to StorageListener & StorageStatusListener, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

@squito
Copy link
Contributor Author

squito commented May 6, 2015

updated version of #4435

  • test file names are sanitized to only use word characters
  • removed giant test file, replaced with a much smaller one

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 6, 2015

Test build #31976 has started for PR 5940 at commit 85fdb3e.

@SparkQA
Copy link

SparkQA commented May 6, 2015

Test build #31976 has finished for PR 5940 at commit 85fdb3e.

  • This patch fails RAT tests.
  • This patch merges cleanly.
  • This patch adds no public classes.

@AmplabJenkins
Copy link

Merged build finished. Test FAILed.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31976/
Test FAILed.

@AmplabJenkins
Copy link

Merged build triggered.

@AmplabJenkins
Copy link

Merged build started.

@SparkQA
Copy link

SparkQA commented May 6, 2015

Test build #31977 has started for PR 5940 at commit 1a72ed6.

@SparkQA
Copy link

SparkQA commented May 6, 2015

Test build #31977 has finished for PR 5940 at commit 1a72ed6.

  • This patch passes all tests.
  • This patch merges cleanly.
  • This patch adds the following public classes (experimental):
    • public class EnumUtil

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/31977/
Test PASSed.

@vanzin
Copy link
Contributor

vanzin commented May 6, 2015

If the changes are only in the test files, looks good to me. nit: pr title has a typo ("separate").

@squito squito changed the title [SPARK-3454] seperate json endpoints for data in the UI [SPARK-3454] separate json endpoints for data in the UI May 6, 2015
pwendell
pwendell reviewed May 8, 2015
View reviewed changes
.rat-excludes
@@ -74,5 +74,12 @@ logs
.*scalastyle-output.xml
.*dependency-reduced-pom.xml
known_translations
json_expectation
local-1422981759269/*
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these timestamps here? If so, maybe this shouldn't be committed to the repo?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah related to test files, nevermind.

@pwendell
Copy link
Contributor

pwendell commented May 8, 2015

Okay verified the filenames are fixed. Going to pull this in now since it's way past the feature deadline.

asfgit pushed a commit that referenced this pull request May 8, 2015
@squito @pwendell
[SPARK-3454] separate json endpoints for data in the UI
532bfda 
Exposes data available in the UI as json over http.  Key points:

* new endpoints, handled independently of existing XyzPage classes.  Root entrypoint is `JsonRootResource`
* Uses jersey + jackson for routing & converting POJOs into json
* tests against known results in `HistoryServerSuite`
* also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

Author: Imran Rashid <irashid@cloudera.com>

Closes #5940 from squito/SPARK-3454_better_test_files and squashes the following commits:

1a72ed6 [Imran Rashid] rats
85fdb3e [Imran Rashid] Merge branch 'no_php' into SPARK-3454
1fc65b0 [Imran Rashid] Revert "Revert "[SPARK-3454] separate json endpoints for data in the UI""
1276900 [Imran Rashid] get rid of giant event file, replace w/ smaller one; check both shuffle read & shuffle write
4e12013 [Imran Rashid] just use test case name for expectation file name
863ef64 [Imran Rashid] rename json files to avoid strange file names and not look like php

(cherry picked from commit c796be7)
Signed-off-by: Patrick Wendell <patrick@databricks.com>
@asfgit asfgit closed this in c796be7 May 8, 2015
@squito
Copy link
Contributor Author

squito commented May 8, 2015

thanks @pwendell !

jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request May 28, 2015
@squito @jeanlyn
[SPARK-3454] separate json endpoints for data in the UI
1658276 
Exposes data available in the UI as json over http.  Key points:

* new endpoints, handled independently of existing XyzPage classes.  Root entrypoint is `JsonRootResource`
* Uses jersey + jackson for routing & converting POJOs into json
* tests against known results in `HistoryServerSuite`
* also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

Author: Imran Rashid <irashid@cloudera.com>

Closes apache#5940 from squito/SPARK-3454_better_test_files and squashes the following commits:

1a72ed6 [Imran Rashid] rats
85fdb3e [Imran Rashid] Merge branch 'no_php' into SPARK-3454
1fc65b0 [Imran Rashid] Revert "Revert "[SPARK-3454] separate json endpoints for data in the UI""
1276900 [Imran Rashid] get rid of giant event file, replace w/ smaller one; check both shuffle read & shuffle write
4e12013 [Imran Rashid] just use test case name for expectation file name
863ef64 [Imran Rashid] rename json files to avoid strange file names and not look like php
@squito squito deleted the SPARK-3454_better_test_files branch June 10, 2015 20:35
jeanlyn pushed a commit to jeanlyn/spark that referenced this pull request Jun 12, 2015
@squito @jeanlyn
[SPARK-3454] separate json endpoints for data in the UI
359baa8 
Exposes data available in the UI as json over http.  Key points:

* new endpoints, handled independently of existing XyzPage classes.  Root entrypoint is `JsonRootResource`
* Uses jersey + jackson for routing & converting POJOs into json
* tests against known results in `HistoryServerSuite`
* also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

Author: Imran Rashid <irashid@cloudera.com>

Closes apache#5940 from squito/SPARK-3454_better_test_files and squashes the following commits:

1a72ed6 [Imran Rashid] rats
85fdb3e [Imran Rashid] Merge branch 'no_php' into SPARK-3454
1fc65b0 [Imran Rashid] Revert "Revert "[SPARK-3454] separate json endpoints for data in the UI""
1276900 [Imran Rashid] get rid of giant event file, replace w/ smaller one; check both shuffle read & shuffle write
4e12013 [Imran Rashid] just use test case name for expectation file name
863ef64 [Imran Rashid] rename json files to avoid strange file names and not look like php
nemccarthy pushed a commit to nemccarthy/spark that referenced this pull request Jun 19, 2015
@squito @nemccarthy
[SPARK-3454] separate json endpoints for data in the UI
9152905 
Exposes data available in the UI as json over http.  Key points:

* new endpoints, handled independently of existing XyzPage classes.  Root entrypoint is `JsonRootResource`
* Uses jersey + jackson for routing & converting POJOs into json
* tests against known results in `HistoryServerSuite`
* also fixes some minor issues w/ the UI -- synchronizing on access to `StorageListener` & `StorageStatusListener`, and fixing some inconsistencies w/ the way we handle retained jobs & stages.

Author: Imran Rashid <irashid@cloudera.com>

Closes apache#5940 from squito/SPARK-3454_better_test_files and squashes the following commits:

1a72ed6 [Imran Rashid] rats
85fdb3e [Imran Rashid] Merge branch 'no_php' into SPARK-3454
1fc65b0 [Imran Rashid] Revert "Revert "[SPARK-3454] separate json endpoints for data in the UI""
1276900 [Imran Rashid] get rid of giant event file, replace w/ smaller one; check both shuffle read & shuffle write
4e12013 [Imran Rashid] just use test case name for expectation file name
863ef64 [Imran Rashid] rename json files to avoid strange file names and not look like php
@JoshRosen JoshRosen mentioned this pull request Apr 6, 2016
Closed
@sarutak sarutak mentioned this pull request Jan 8, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@squito @AmplabJenkins @SparkQA @vanzin @pwendell