Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Jaeger agent gets stuck when closing with SocketBufferSize set #4448

Closed
ChenX1993 opened this issue May 9, 2023 · 0 comments · Fixed by #4449
Closed

[Bug]: Jaeger agent gets stuck when closing with SocketBufferSize set #4448

ChenX1993 opened this issue May 9, 2023 · 0 comments · Fixed by #4449
Labels

Comments

@ChenX1993
Copy link
Contributor

ChenX1993 commented May 9, 2023

What happened?

Problem

With SocketBufferSize set in config, the jaeger-agent gets stuck when closing.

SocketBufferSize int `yaml:"socketBufferSize"`

Root cause

This is because Close()of net.UDPConn will be blocked if Fd() is used to get the file descriptor.

The place to set the SocketBufferSize in jaeger-agent is below :

func setSocketBuffer(conn *net.UDPConn, bufferSize int) error {
file, err := conn.File()
if err != nil {
return err
}
return syscall.SetsockoptInt(int(file.Fd()), syscall.SOL_SOCKET, syscall.SO_RCVBUF, bufferSize)
}

And the implementation of Fd():

func (f *File) Fd() uintptr {
	if f == nil {
		return ^(uintptr(0))
	}

	// If we put the file descriptor into nonblocking mode,
	// then set it to blocking mode before we return it,
	// because historically we have always returned a descriptor
	// opened in blocking mode. The File will continue to work,
	// but any blocking operation will tie up a thread.
	if f.nonblock {
		f.pfd.SetBlocking()
	}

	return uintptr(f.pfd.Sysfd)
}

The exact same problem has been discussed here: golang/go#29277

Will create PR for fix.

Steps to reproduce

  1. Set SocketBufferSize with any valid number
  2. Run agent
  3. Stop agent

This issue can also reproduce in unit test:
e25216c

Expected behavior

Jaeger agent can stop successfully.

Relevant log output

No response

Screenshot

No response

Additional context

No response

Jaeger backend version

latest

SDK

without sdk

Pipeline

No response

Stogage backend

No response

Operating system

Linux, Mac

Deployment model

No response

Deployment configs

No response

@ChenX1993 ChenX1993 added the bug label May 9, 2023
yurishkuro pushed a commit that referenced this issue May 11, 2023
<!--
Please delete this comment before posting.

We appreciate your contribution to the Jaeger project! 👋🎉

Before creating a pull request, please make sure:
- Your PR is solving one problem
- You have read the guide for contributing
- See
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING.md
- You signed all your commits (otherwise we won't be able to merge the
PR)
- See
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md#certificate-of-origin---sign-your-work
- You added unit tests for the new functionality
- You mention in the PR description which issue it is addressing, e.g.
"Resolves #123"
-->

## Which problem is this PR solving?
Resolves #4448

## Short description of the changes
Jaeger agent gets stuck when closing with SocketBufferSize set. This is
because `Close()` of `net.UDPConn` will be blocked if `Fd()` is used to
get the file descriptor.
Use `RawConn.Control` instead to get fd to set the socket buffer.

Same issue was discussed here: golang/go#29277
The fix refers to here: brucespang/go-tcpinfo#3

Signed-off-by: Chen Xu <chen.x@uber.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant