Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Support the asyncio debug mode and apply aiomonitor-ng #688

Merged
merged 10 commits into from Aug 29, 2022

Conversation

achimnol
Copy link
Member

@achimnol achimnol commented Aug 25, 2022

If the asyncio debug mode is enabled, it additionally keeps track of the stack trace of where the tasks are created though this behavior is undocumented.

Chaining of full task creation/cancellation tracebacks is handled in our modified aiomonitor.

This PR also fixes missing sample configurations for aiomonitor ports and changes the default value of them to avoid overlapping of the aiomonitor ports in manager and agent when they run in a same host (e.g., development setups).

@achimnol achimnol added this to the 22.03 milestone Aug 25, 2022
@achimnol achimnol added type:feature Add new features area:infrastructure Infrastructure-related issues labels Aug 25, 2022
@achimnol achimnol self-assigned this Aug 25, 2022
@github-actions github-actions bot added comp:agent Related to Agent component comp:manager Related to Manager component comp:storage-proxy Related to Storage proxy component labels Aug 25, 2022
@fregataa
Copy link
Contributor

the port number of aiomonitors changes due to better management of ports?

@achimnol
Copy link
Member Author

achimnol commented Aug 26, 2022

the port number of aiomonitors changes due to better management of ports?

Manager and agent's aiomonitor ports were overlapping when there are two or more worker processes in the manager (the actual aiomonitor port used by each worker process = the configured port number + pidx). Unfortunately in development setups this did not cause any explicit error because all these processes run in the same user privilege and the SO_REUSEPORT option. In production, we are already allocating the same "more distant" port numbers.

@achimnol achimnol changed the title feat: Enable the asyncio debug mode in our debug mode feat: Support the asyncio debug mode and apply aiomonitor-ng Aug 26, 2022
@achimnol achimnol requested review from adrysn and xyloon August 26, 2022 09:39
- The "creation location" in the ps command output was wrong because
  0.5.0 retrieved the other end of the stack.
Copy link
Member

@adrysn adrysn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It works as intended in my environment. LGTM.

@achimnol
Copy link
Member Author

I will make separate PRs for future updates of aiomonitor-ng.

@achimnol achimnol merged commit c22a89f into main Aug 29, 2022
@achimnol achimnol deleted the feature/set-asyncio-debug-mode branch August 29, 2022 15:29
achimnol added a commit that referenced this pull request Aug 30, 2022
* feat: Add `[debug].{asyncio,enhanced-aiomonitor-task-info}` config flags
  in the manager, agent, and storage-proxy TOML files
* fix: Prevent overlapping of manager/agent aiomonitor ports by changing
  the default ports to have more distant values (50100, 50200).
* setup: Replace aiomonitor with aiomonitor-ng
* feat: Add aiomonitor support to storage-proxy (default port: 50300)

Backported-From: main
Backported-To: 22.03
@Yaminyam Yaminyam added the size:XL 500~ LoC label Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area:infrastructure Infrastructure-related issues comp:agent Related to Agent component comp:manager Related to Manager component comp:storage-proxy Related to Storage proxy component size:XL 500~ LoC type:feature Add new features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants