Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat: support container root filesystem log collection #209

Merged
merged 1 commit into from
Jun 9, 2022

Conversation

ethfoo
Copy link
Collaborator

@ethfoo ethfoo commented May 24, 2022

Proposed Changes:

enable this feature by rootFsCollectionEnabled see below:

  discovery:
    enabled: true
    kubernetes:
      rootFsCollectionEnabled: true
  • docker runtime: only support overlay2 graph driver, retrieve the node path in merged rootfs by inspecting the docker status
  • containerd runtime: get pid by containerId, then find the path in /proc/{pid}/root

Which issue(s) this PR fixes:

Fixes #208

Additional documentation:

  • TODO

@ethfoo ethfoo changed the title Feat: support container root filesystem log collection WIP Feat: support container root filesystem log collection May 24, 2022
@ethfoo ethfoo force-pushed the feat-proc-root branch 4 times, most recently from 497af4e to 7fa7e1d Compare May 25, 2022 02:15
@ethfoo ethfoo changed the title WIP Feat: support container root filesystem log collection Feat: support container root filesystem log collection May 25, 2022
Copy link
Collaborator

@iamyeka iamyeka left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reviewed

pkg/discovery/kubernetes/helper/kube.go Show resolved Hide resolved
@@ -386,33 +388,38 @@ func updatePaths(config *Config, s *fileSource, pod *corev1.Pod, containerName,
return nil
}

func getPathsInNode(config *Config, containerPaths []string, pod *corev1.Pod, containerName string, containerId string) ([]string, error) {
func (c *Controller) getPathsInNode(config *Config, containerPaths []string, pod *corev1.Pod, containerName string, containerId string) ([]string, error) {
if len(containerPaths) == 0 {
return nil, errors.New("path is empty")
}

var paths []string
for _, p := range containerPaths {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为什么不把这段逻辑放在下面helper.PathsInNode方法的实现里,我看PathsInNode里面还特意忽略掉了path为stdout的情况,而且GenContainerStdoutLog这个方法和PathsInNode也是在一个包里面,直接调用应该也比较方便

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

continue
}
}

p, err := helper.PathsInNode(config.KubeletRootDir, containerPaths, pod, containerName)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里是如果containerPaths中通过volume方式找不到路径就失败了,然后全部检测是否是rootfs,有没有可能用户配了volume格式的path,也配了rootfs,两种path都有?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果找不到volume,会去找rootfs,不冲突

@@ -71,6 +83,8 @@ func (d *Discovery) Start(stopCh <-chan struct{}) {
logConfInformerFactory.Loggie().V1beta1().LogConfigs(), logConfInformerFactory.Loggie().V1beta1().ClusterLogConfigs(), logConfInformerFactory.Loggie().V1beta1().Sinks(),
logConfInformerFactory.Loggie().V1beta1().Interceptors(), nodeInformerFactory.Core().V1().Nodes())

ctrl.Runtime = d.runtime
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

controller其他参数都通过NewController方法传递,为什么Runtime要通过赋值的方式来传递?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里传参太多了,后面可能会重构一下

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

还是改用传参了

func Init(endpoints []string, defaultRuntime string) Runtime {
runtimeName := defaultRuntime
if defaultRuntime == "" {
runtimeName = getRuntimeName(endpoints, defaultRuntime)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里defaultRuntime已经是空了,还需要作为参数往下传吗?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

pkg/discovery/kubernetes/controller/config.go Show resolved Hide resolved
pkg/discovery/kubernetes/controller/config.go Show resolved Hide resolved
pkg/discovery/kubernetes/runtime/runtime.go Show resolved Hide resolved
continue
}

rootfsPaths = append(rootfsPaths, rootfsPrePath+p)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

p这个path是/开头吗,我看rootfsPrePath以merged结尾的,会不会缺/的情况?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

我改成path.join吧

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

if p == logconfigv1beta1.PathStdout {
continue
}
rootfsPaths = append(rootfsPaths, prefix+p)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

同docker实现的comment

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

@ethfoo ethfoo force-pushed the feat-proc-root branch 2 times, most recently from a854ce4 to 9d3954c Compare June 9, 2022 05:55
@iamyeka iamyeka merged commit 23a7552 into main Jun 9, 2022
@ethfoo ethfoo deleted the feat-proc-root branch June 10, 2022 12:52
evanzhang87 pushed a commit to evanzhang87/loggie that referenced this pull request Aug 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Log collection of container file system
2 participants