[WIP] monitor container termination status info when getting pod info

[jrafanie]

WIP...

This is what it looks like in pp MiqServer.my_server.worker_manager.current_pods

...
 "1-generic-7f6466d498-n2xh9"=>
  {:label_name=>"1-generic",
   :container_restarts=>1,
   :last_state_terminated=>true,
   :terminations=>
    [{:container_id=>
       "cri-o://c2f9c6a0d11c8befc0dfa1431a9ce8f73b22d39526b839f4d15b865d2ce46f10",
      :exit_code=>137,
      :message=>nil,
      :reason=>"OOMKilled",
      :signal=>nil,
      :started_at=>"2022-12-02T13:51:23Z",
      :finished_at=>"2022-12-02T15:22:07Z"}]},
 "1-google-cloud-event-catcher-4-596d898775-fqvf8"=>
  {:label_name=>"1-google-cloud-event-catcher-4",
   :container_restarts=>0,
   :last_state_terminated=>false,
   :terminations=>[]},
...

[jrafanie]

The alternative way of doing this figuring out what termination events look like and monitor just those events.

See lines 178-195 above.

[jrafanie]

I chose to clear the array each time through. The server code that checks the current_pods concurrent hash can possibly miss terminations or deal with things it already processed... so we might want to just monitor for termination events as mentioned 👇 . I'm not sure how much work that is.

[Fryguy]

really looking forward to this...thanks @jrafanie

[jrafanie]

I also found this but it assumes you can install things in the cluster so at best, we could possibly suggest it: https://medium.com/@andrew.kaczynski/kubernetes-events-how-to-keep-historical-data-of-your-cluster-835d685cc45

[miq-bot]

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s)

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[miq-bot]

This pull request has been automatically closed because it has not been updated for at least 3 months.

Feel free to reopen this pull request if these changes are still valid.

Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.

[miq-bot]

Checked commit jrafanie@0aaf05b with ruby 2.6.10, rubocop 1.28.2, haml-lint 0.35.0, and yamllint
1 file checked, 1 offense detected

app/models/miq_server/worker_management/kubernetes.rb

• ❗ - Line 224, Col 7 - Style/Next - Use next to skip iteration.

[miq-bot]

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s).

[miq-bot]

This pull request has been automatically marked as stale because it has not been updated for at least 3 months.

If these changes are still valid, please remove the stale label, make any changes requested by reviewers (if any), and ensure that this issue is being looked at by the assigned/reviewer(s).