Skip to content

java: Paranoid profiling stop on suspicious events in kernel messages #235

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Dec 1, 2021
Merged

java: Paranoid profiling stop on suspicious events in kernel messages #235

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
6d53bb7
java: Paranoid profiling stop on suspicious events in kernel messages
Jongy Nov 30, 2021
ac72950
java: Log about profiling stop just once
Jongy Dec 1, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 25 additions & 8 deletions gprofiler/profilers/java.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -493,8 +493,9 @@ def __init__(

@classmethod
def _disable_profiling(cls):
logger.warning("Java profiling has been disabled, will avoid profiling any new java process")
cls._should_profile = False
if cls._should_profile:
logger.warning("Java profiling has been disabled, will avoid profiling any new java process")
cls._should_profile = False

def _is_jvm_type_supported(self, java_version_cmd_output: str) -> bool:
return all(exclusion not in java_version_cmd_output for exclusion in self.JDK_EXCLUSIONS)
Expand Down Expand Up @@ -755,15 +756,31 @@ def _proc_exit_callback(self, tid: int, pid: int, exit_code: int):
def _handle_kernel_messages(self, messages):
for message in messages:
_, _, text = message
entry = get_oom_entry(text)
if entry and entry.pid in self._profiled_pids:
logger.warning("Profiled Java process OOM", oom=json.dumps(entry._asdict()))
oom_entry = get_oom_entry(text)
if oom_entry and oom_entry.pid in self._profiled_pids:
logger.warning("Profiled Java process OOM", oom=json.dumps(oom_entry._asdict()))
self._disable_profiling() # paranoia
continue

entry = get_signal_entry(text)
if entry and entry.pid in self._profiled_pids:
logger.warning("Profiled Java process signalled", signal=json.dumps(entry._asdict()))
signal_entry = get_signal_entry(text)
if signal_entry is not None and signal_entry.pid in self._profiled_pids:
logger.warning("Profiled Java process signalled", signal=json.dumps(signal_entry._asdict()))
self._disable_profiling() # paranoia
continue

if self._java_safemode:
Copy link
Contributor

@d3dave d3dave Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if java safemode and profiling enabled

Copy link
Contributor Author

@Jongy Jongy Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is on purpose actually, The system-wide things - I want to continue collecting them also after profiling was stopped (so I can understand whether it was an anomaly or not)

Copy link
Contributor Author

@Jongy Jongy Dec 1, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't have profiled PIDs anymore, that's true

if oom_entry is not None:
logger.warning("General OOM", oom=json.dumps(oom_entry._asdict()))
elif signal_entry is not None:
logger.warning("General signal", signal=json.dumps(signal_entry._asdict()))
elif any(str(p) in text for p in self._profiled_pids):
logger.warning("Profiled PID shows in kernel message line", line=text)
else:
continue

# paranoia - stop Java profiling upon any OOM / fatal-signal / occurence of a profiled
# PID in a kernel message.
self._disable_profiling()

def _handle_new_kernel_messages(self):
try:
Expand Down