[Scaling Up]Runner created but not registered in github

[wippyz]

Hi we suddenly have problem where the scaled up runner are created successfully but not registered in github (not showing in the runner menu in github)
would appreciate any help to where to start troubleshooting it.
Thank you very much

[npalm]

Two options for quick checks

• look up the logs in cloudwatch
• in ec2, select instance, select connect, select ssm. Next check the logs in /vag/log

[mahela-aws]

I'm also experiencing a similar issue when I change the ami to ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-* but works fine with amzn2-ami-hvm-2.*-x86_64-gp2 ami.

I will try to check /var/logs as suggested above, I'm curious if this has been tested with ubuntu images @npalm

[mahela-aws]

also I cannot see anything suspicious in cw logs, scaling up and down works just fine but runners are not registered in github organisation

[wippyz]

i couldnt find anything suspicious in /var/logs ... will try to take a look at the ami

[ScottGuymer]

The userdata script will log to /var/log/user-data.log you should see at least something in this log.

If not then its likely that the userdata functionality is not working as expected and the script is not being triggered.

Do you customise the userdata script in any way?

[mahela-aws]

@ScottGuymer we are doing this, but i dont think we are touching any user-data scripts

# We need to remap the docker user (to a non-root one) because we otherwise
# might leave files behind that cannot be manipulated.
# This would break all subsequent jobs, specifically the checkout step.
userdata_post_install = <<EOT
sudo echo dockremap:1000:1000 > /etc/subuid
sudo echo dockremap:1000:1000 > /etc/subgid
cat << EOF >> /etc/docker/daemon.json
{
  "userns-remap": "default"
}
EOF
sudo service docker restart
EOT

but anyway could ☝️ be a reason for ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-* to not function properly(not getting registered with github) ?

[mahela-aws]

this is what I see in user-data.log

ssm-user@ip-10-0-1-93:/var/snap/amazon-ssm-agent/4046$ cat /var/log/user-data.log
/var/lib/cloud/instance/scripts/part-001: line 6: yum: command not found

[npalm]

Are you using amazon linux2?

[ScottGuymer]

If you are using Ubuntu or some other debian derivative you need to customise the user-data script something like this example

https://github.com/philips-labs/terraform-aws-github-runner/blob/develop/examples/ubuntu/templates/user-data.sh

[mahela-aws]

yeah I noticed it now.
https://github.com/philips-labs/terraform-aws-github-runner/blob/develop/modules/runners/templates/user-data.sh
this script is not compatible with ubuntu

[mahela-aws]

@ScottGuymer if we want to switch between ubuntu and amzn-linux runners how can we achieve this ? we are referencing your module in our code base

[mahela-aws]

by default is it using amzn-linux compatible user-data script ?

[ScottGuymer]

By default it assumes amzn linux but you can look at the ubuntu example for a way run unbuntu agents..

[scream314]

What do you think about adding the example Ubuntu scripts to modules/runners/templates/ as ${original_name}-ubuntu.sh, adding 1 additional key (eg. linux-ubuntu) to each of the locals default_ami , default_userdata_template, userdata_install_runner and userdata_start_runner, so one could use the variable runner_os (eg. set it to linux-ubuntu) to control what kind of runner is created.

I think adding more support for Ubuntu would be great, as

• there are already examples for building an Ubuntu-based AMI
• many of the public actions do not really support AMZ2

[ScottGuymer]

I have opened #2022 to consider moving fully to ubuntu for the module.

Closing this to have further discussion there.