Build failed likely due to unclean state of repo for self-hosted runner

[potiuk]

This looks like a failure because of previously cancelled build on the self-hosted runner:

https://github.com/apache/airflow/runs/1993357873?check_suite_focus=true#step:2:103

Looks like the repository was attempted to be checked out when there was a source code already in the folder where the source code should be checked out.

Some 'egg-info" files are there apparently created by the previous run. It would be great to run equivalent of:
sudo rm -rf SOURCE_DIR && mkdir -pv SOURCE_DIR - always in the runner to avoid this kind of problem.

---

Copy of the output.

Cleaning the repository
  /usr/bin/git clean -ffdx
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/SOURCES.txt: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/top_level.txt: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/requires.txt: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/entry_points.txt: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/dependency_links.txt: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/not-zip-safe: Permission denied
  warning: failed to remove provider_packages/apache_airflow_providers_microsoft_azure.egg-info/PKG-INFO: Permission denied
  warning: failed to remove provider_packages/build/bdist.linux-x86_64: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/transfers/oracle_to_azure_data_lake.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/transfers/local_to_adls.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/transfers/file_to_wasb.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/transfers/azure_blob_to_gcs.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/transfers/__init__.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/sensors/wasb.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/sensors/azure_cosmos.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/sensors/__init__.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/secrets/azure_key_vault.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/secrets/__init__.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/wasb_delete_blob.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/azure_cosmos.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/azure_container_instances.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/azure_batch.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/adx.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/adls_list.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/adls_delete.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/operators/__init__.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/log/wasb_task_handler.py: Permission denied
  warning: failed to remove provider_packages/build/lib/airflow/providers/microsoft/azure/log/__init__.py: Permission denied

[potiuk]

Similar problem here: https://github.com/apache/airflow/pull/14531/checks?check_run_id=1999125831

[potiuk]

Also here: https://github.com/apache/airflow/pull/14532/checks?check_run_id=1999112392

[potiuk]

This is almost for sure result of an unclean directory in one of the runners which was cancelled in one of the previous builds.

[potiuk]

cc: @ashb

[ashb]

Will take a look in detail tomorrow, but permissions are being reset by https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.

Wonder what went on here

[potiuk]

Will take a look in detail tomorrow, but permissions are being reset by https://github.com/apache/airflow-ci-infra/blob/user-data-WIP/cloud-init.yml#L129.

Wonder what went on here

Is the chown run always before ? And with root user ?

I think this is one of those cases where I prefer to have a completely "pristine" empty directory rather trying to reuse last checkout.

GitHub checkout by default only fetches one single commit, it is super-fast so we save basically nothing by caching the result of previous checkout. And we avoid any kind of errors like this one - where some edge case we missed happens.

I had in the past numerous similar issues on pretty much every single CI system I built. And starting from clean state was always final state I got to. This saved a lot of time for me and for the users trying to understand what's going on.

I propose we just hard-wipe workdir always. We will save a lot of time for debugging - this is what pretty much every single CI out there does, so there is no point in doing it differently.

[potiuk]

My hypothesis is that the previous runs docker containers are still working in the background (killing a docker container running in the background might not happen immeddiately). Those running docker containers could continue creating files while the next "init" is happening. That's why "chown" would not work - because the files get created .

That's where re-creating the whole tmpfs from scratch could help (if that hypothesis is right) - first of all deleting the tmpfs volume could fail (or wait until the docker container finish), secondly recreating the whole volume is a sure way to always be 'clean' - even if there is anything writing to the old tmpfs, the new volume will not share anything with the new one.

[ashb]

Is the chown run always before ?

Yes. That is in the pre-start script, and the runner is configured to run a single job then stop (at which point systemd restarts it)

And with root user ?

Yes otherwise it wouldn't do anything

[ashb]

Collecting info/making notes about this as I debug:

• Job that ran on this host just before https://github.com/apache/airflow/runs/1993313279?check_suite_focus=true
• Previous job on instance was cancelled. Relevant?
• When the job in question failed, it did correctly clean up and reset the permissions (I see 2021-02-27 09:27:55.586,changed ownership of './provider_packages/build' from root:root to runner:runner in the logs for instance)

[ashb]

My hypothesis is that the previous runs docker containers are still working in the background (killing a docker container running in the background might not happen immediately).

I think you are right. I was fixing the permissions and cleaning up the files, and then deleting running docker containers. That needs reversing.