HADOOP-18055. Async Profiler endpoint for Hadoop daemons #3824
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This comment has been minimized.
This comment has been minimized.
|
@aajisaka @jojochuang @tasanuma Could you please take a look? |
This comment has been minimized.
This comment has been minimized.
|
Thank you @virajjasani for your patch. This feature looks very helpful for us to investigate the root cause of some performance issues. I'll build and deploy Hadoop cluster and test this feature. Now I have one comment: We need to add an additional access control for this servlet like ConfServlet. In addition, we need to update the document of |
virajjasani
commented
Dec 24, 2021
| @Override | ||
| protected void doGet(final HttpServletRequest req, final HttpServletResponse resp) | ||
| throws IOException { | ||
| if (!HttpServer2.isInstrumentationAccessAllowed(getServletContext(), req, resp)) { |
There was a problem hiding this comment.
Thanks @aajisaka. I have added isInstrumentationAccessAllowed here but let me also add it to ProfileOutputServlet so that even before the url redirection takes place, user will get SC_UNAUTHORIZED response.
aajisaka
reviewed
Dec 24, 2021
Comment on lines
58
to
60
| * If Namenode http address is 9870, and ResourceManager http address is 8088, | ||
| * ProfileServlet running with async-profiler setup can be accessed with | ||
| * http://localhost:9870/prof and http://localhost:8088/prof |
There was a problem hiding this comment.
Should we comment the NameNode and ResourceManager is running on localhost?
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
|
🎊 +1 overall
This message was automatically generated. |
jojochuang
reviewed
Jan 4, 2022
| HttpURLConnection conn = | ||
| (HttpURLConnection) new URL(baseUrl, "/prof").openConnection(); | ||
| assertEquals("GET", conn.getHeaderField(ProfileServlet.ACCESS_CONTROL_ALLOW_METHODS)); | ||
| assertNotNull(conn.getHeaderField(ProfileServlet.ACCESS_CONTROL_ALLOW_ORIGIN)); |
There was a problem hiding this comment.
would it be a good idea to call conn.disconnect() to clean up?
| private static final int DEFAULT_DURATION_SECONDS = 10; | ||
| private static final AtomicInteger ID_GEN = new AtomicInteger(0); | ||
|
|
||
| static final String OUTPUT_DIR = System.getProperty("java.io.tmpdir") + "/prof-output"; |
There was a problem hiding this comment.
because the ProfileServlet is used by several systems (Ozone, HBase, Hive), I would suggest to make the default output dir specific to Hadoop
There was a problem hiding this comment.
Makes sense, updated. (good to know Ozone also supports this)
Yes, updated AsyncProfilerServlet.md. Thanks |
|
🎊 +1 overall
This message was automatically generated. |
jojochuang
reviewed
Jan 4, 2022
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Reviewed-by: Akira Ajisaka <aajisaka@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Async profiler (https://github.com/jvm-profiling-tools/async-profiler) is a low overhead sampling profiler for Java that does not suffer from Safepoint bias problem. It features HotSpot-specific APIs to collect stack traces and to track memory allocations. The profiler works with OpenJDK, Oracle JDK and other Java runtimes based on the HotSpot JVM.
Async profiler can also profile heap allocations, lock contention, and HW performance counters in addition to CPU.
This PR provides Async profiler endpoint for Hadoop daemons.
More analysis on CPU and memory overhead:
How was this patch tested?
Locally. Screenshots:
For code changes: