Add log watch entry points

[akiselev1]

Adding ironic and ironic-inspector log watch entry points per this design proposal: https://github.com/metal3-io/metal3-docs/blob/master/design/ironic-debuggability-improvement.md

[akiselev1]

/cc stbenjam

[dhellmann]

/cc @kashifest @maelk

/test-integration

[stbenjam]

/approve

[metal3-io-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: akiselev1, stbenjam

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [stbenjam]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dhellmann]

/test-integration
/lgtm

[dhellmann]

/approve
/lgtm cancel

Oops, I meant to give @maelk and @kashifest a chance to review this.

[kashifest]

I am not sure if we would want to add the container here also https://github.com/metal3-io/baremetal-operator/blob/master/tools/run_local_ironic.sh since in case of test integration we deploy the ironic locally. When this is decided, I would also like to run centos tests to verify ironic deployment.

[maelk]

+1 to adding it to the run_local_ironic.sh script. Otherwise lgtm

[maelk]

btw, if it is added to run_local_ironic.sh, the cleanup scripts in metal3-dev-env will need to be updated too.

[akiselev1]

btw, if it is added to run_local_ironic.sh, the cleanup scripts in metal3-dev-env will need to be updated too.

@maelk Can you share the link to these cleanup scripts?

[akiselev1]

I am not sure if we would want to add the container here also https://github.com/metal3-io/baremetal-operator/blob/master/tools/run_local_ironic.sh since in case of test integration we deploy the ironic locally. When this is decided, I would also like to run centos tests to verify ironic deployment.

@kashifest What is the command to run centos tests?

[maelk]

https://github.com/metal3-io/metal3-dev-env/blob/master/lib/common.sh#L426

The command would be / test-centos-integration

[akiselev1]

/test-centos-integration

[akiselev1]

/test-integration

[akiselev1]

/test-integration

[akiselev1]

/test-integration

[fmuyassarov]

I'm not 100% sure if that's the reason for the below issue on the failed CI

+ sudo docker run -d --net host --privileged --name ironic-log-watch --entrypoint /bin/runlogwatch.sh -v /opt/metal3-dev-env/ironic:/shared 192.168.111.1:5000/localimages/ironic
4e7fd9e40796358daacc433d3bbd983c745f2da64b79ac6028f10fb1ff01226a
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "exec: \"/bin/runlogwatch.sh\": permission denied": unknown.
make: *** [Makefile:10: launch_mgmt_cluster] Error 126

but it seems that runlogwatch.sh is not executable (no +x permission).
I ran this container locally, and went inside and I saw that runlogwatch.sh doesn't have +x permission. See below

fmuyassarov@est:~$ docker ps
CONTAINER ID        IMAGE                      COMMAND                  CREATED             STATUS              PORTS               NAMES
72f75fae68a9        quay.io/metal3-io/ironic   "/bin/runironic --en…"   12 seconds ago      Up 11 seconds                           nervous_shtern
fmuyassarov@est:~$ docker exec -it 72 /bin/bash
[root@72f75fae68a9 /]# ls -lah /bin/ | grep runlogwatch.sh
-rw-r--r-- 1 root root   389 Nov 12 10:57 runlogwatch.sh
[root@72f75fae68a9 /]# 

[akiselev1]

@fmuyassarov Thank you! I think I know what happened. I forgot to set +x permission in the original PR, and created a new one: metal3-io/ironic-image#221 to fix that. You can see correct +x permission if you do git clone https://github.com/metal3-io/ironic-image.git. Apparently that did not re-build the image in Quay (why?). I guess another ironic-image PR with some minor change in Dockerfile would push this change to Quay. Is there any better/smarter way to re-build the container and push it to Quay? cc: @maelk , @dhellmann

[fmuyassarov]

Make sense. Last built in the quay for the Ironic image was on Noveber 12th, 13:02 (I'm not sure though if it shows my local time)

and from the git log your fix was merged on November 12th, at 14:02 (it is my local time). Maybe there is some time mismatch on my browser. I would assume quay was triggered, but then ...why didn't it take changes...

Merge: 749dec4 47ec04f
Author: metal3-io-bot <55852648+metal3-io-bot@users.noreply.github.com>
Date:   Thu Nov 12 14:02:08 2020 -0500

    Merge pull request #221 from akiselev1/fix-script-permissions
    
    Add runlogwatch.sh +x permission

But hopefully Doug or Maël have some ideas. Maybe there is a way we can trigger the build for the quay?

[akiselev1]

Looks like we hit an iceberg: You have reached your pull rate limit.

[akiselev1]

/test-integration

[fmuyassarov]

Something went wrong in the CI infra I believe.
/test-integration

[akiselev1]

Hm... Ironic image did get updated in Quay. But CI results are empty. Do we have any doc on metal3 CI architecture? Let me know if I can help in any way.

[kashifest]

it seems like the opensuse pages are down. The CI failed to download the release keys.

[akiselev1]

Opensuse is back.
/test-integration

[akiselev1]

/cc @fmuyassarov @maelk @kashifest - CI finally passed. May I have lgtm from one of you?

[kashifest]

/lgtm
/test-v1a3-integration
/test-v1a3-centos-integration
Just to be absolutely sure, will run one more round of CI with v1a3

[maelk]

/test-v1a3-centos-integration