Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Eru-core loses track of running Lambda when eru-core is restarted #279

Closed
EastonLee opened this issue Nov 26, 2020 · 1 comment
Closed
Assignees

Comments

@EastonLee
Copy link
Contributor

Eru-core tracks the running of Lambda, but when eru-core is restarted it loses the track of running Lambdas, this could cause long-running Lambda container leak and never get cleaned.

@CMGS
Copy link
Contributor

CMGS commented Nov 26, 2020

这个问题是这样的:

  1. 我们现有的机制只保证了 grpc 请求的可靠性,grpc 请求在,core 除非 kill -9, 否则正常退出是不会导致 leak
  2. lambda 在 background run 的时候 grpc 请求已经没得了
  3. 这时候 someone 更新 core,通过 core 保证 background workload recycle 和 expire recycle 失效
  4. 这个 lambda 容器就永久存在了

解决方案的话我能想到的是:

  1. 每个 core 在退出的时候通过 hostname 作为 path 记录一个状态到 store 里面
  2. 这个 host 上的 core 启动的时候自检,load 状态做对应处理,然后删除状态
  3. 每个请求进来的时候记录一个状态,请求结束的时候清理,同样也需要在 core 启动的时候进行自检

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants