Add ssh debugging for Github Actions CI.

[pratikvn]

This PR adds ssh debugging capability to the github actions builds. This is done by creating a ssh connection to the machine with the https://github.com/marketplace/actions/debugging-with-tmate action.

This can be really helpful to debug Windows and OSX builds for us when we dont have access to Windows/OSX machines.

Currently, once this PR has been merged into develop (our default branch), any branch that is based on develop will be able to use this ssh debugging capability.

For example, let's say that a develop branch already contains this PR merged, if I create a new branch called debug-ci and push it and see that it fails for some build, you can go to Actions -> <Specific workflow (OSX/Windows)> on the left graph and you will see something like this:

Here you are manually triggering the workflow and if you enter debug_enabled in the box and say run workflow, it will run the Debug over SSH (tmate) job and spit out a ssh server@domain to which you can directly ssh to. An example view for the OSX job:

To make things easier with the Windows jobs, I also split them up into Windows-{MSVC-CUDA, Reference, MinGW, CygWin}, so that you can only manually trigger the ones you need.

[yhmtsai]

LGTM in general.
I think we need to use ssh-key, or I guess anyone can go into the ssh and get the environment variable.
Could you check whether the ssh can get the token or some variables of Github secret?

[yhmtsai]

one more question: do you know how they finish the job?
exceeding the github action time limit, or logout of ssh can finish the job

[pratikvn]

I checked if the environment/repository secrets are leaked and it does not seem to be the case. But we can also set

with:
        limit-access-to-actor: true

for the step and allow ssh access only with a ssh-key. But I did not see any need for this right now, but I am open to adding this.

You can finish the job by logging out or by doing a touch continue in the project root directory.

By default, it is the github actions timeout limit, but we can also set a timeout limit manually, if we want to as well. But I think the github actions timeout limit is sufficient for us.

[upsj]

As far as I have seen so far, the output logs are restricted in visibility, so if I'm logged out, I can't see the ssh connection information. I guess that's limited to the organization members or similar?

[upsj]

Is there something we can do about the ordering of the jobs? After the separation into different files, they are all over the place 😄

[pratikvn]

Yes, the logs should be restricted to collaborators only Edit: Logs are visible to all signed in users. Also the manual triggers for the debug builds can only be done by people with write access to the repo.

Regarding the job ordering, I am not sure how to fix that. It seems to be ordered differently everytime I see it and depending on which jobs are running :)

[yhmtsai]

@pratikvn that's good. Thanks for checking.
the visibility of log only for collaborators should be good enough

[pratikvn]

Regarding the ordering of the jobs, it seems to be alphabetical, atleast when the jobs are the same state (pending, running, completed or failed) and in that order between states. I could rename all jobs to have a number before them. So, they would look like 1-OnNewPR... , 2-OnSyncPR , 3-OSX... and so on. But I am not sure it is necessary because after completion, the jobs should have a reasonable order.

[pratikvn]

So, after some checking and verification with @yhmtsai , we discovered the following:

1. The debug through ssh (which is a manual trigger), can only be done by collaborators.
2. Logs are visible to everyone who is logged in to github. Therefore, if someone triggers a debug-ssh build, and the ssh connection link is being spit out, everyone who is logged in to github, will be able to see it. They will also be able to ssh into the connection.
3. If the workflow does not have any secrets, then these are not leaked in the ssh session. Therefore, if this is being used, we must not use any secrets in the workflow.

Given the above, I think it is still reasonably secure to use this in our Windows and OSX builds because:

1. We do not use any secrets in these workflows.
2. The manual triggering of the builds provides an additional layer of security so that only collaborators are able to trigger these workflows.

If anyone still has some concerns, please let me know.

P.S: I tried to add another layer of security with the requirement of allowing ssh connections to only those in the collaborator list by trying to limit the access to actor. But I am having some weird rate limiting issues with that. We can probably get back to this at a later point.

[upsj]

In that case, I would prefer if we used limit-access-to-actor: true, since (I guess) we all have SSH keys setup in our Github accounts?

[pratikvn]

So, the root of the rate limiting issue seems to be that Github actions currently has very few MacOS machines, probably all with the same IP address. For this reason, you face issues when trying to connect via ssh with a public key, because github has rate limitations based on IP. This issue does not seem to occur for the Windows and Linux machines.

So, I added the ssh + public_key requirement for the windows jobs, but left the non-public key for the OSX job for now. If they fix the issues on their end, we should be able use limit-access-to-actor for OSX jobs as well. For now, I think it should be okay to use them without the restriction of a public_key.

[pratikvn]

Just to document the issue with OSX. It has been observed by others as well. mxschmitt/action-tmate#69

Here the issue reporter mentions that the problem might be that the actions CI has limited Mac machines and hence runs into rate limiting issues. I don't really want to pass the GITHUB_TOKEN variable through, so for now, I leave it as it is and in the future if the issue gets resolved from github actions, we can add back the limit-access-to-actor for the OSX jobs as well.