teuthology/suite/run.py: Improve ScheduleFail exception

[kamoltat]

As per: ceph/pulpito-ng#23

1. Improve teuthology-schedule exceptions by utilizing ScheduleFail and GitError
2. make teuthology.suite.main return job_count such that teuthology-api and Pulpito-ng can use it.
3. edit teuthology.sh & start.sh such that we -x earlier for better output when watching the docker-compose do it's thing.

Fixes: https://tracker.ceph.com/issues/64820

[kamoltat]

Fixes: https://tracker.ceph.com/issues/64820

[zmc]

Just a couple minor changes suggested; have you scheduled a run with this branch? I'd be curious to see it if so

[kamoltat]

integration test failed:

Warning: RNING]: Skipping Galaxy server https://galaxy.ansible.com./ Got an unexpected
86.80 error when getting available versions of collection amazon.aws: Unknown error
86.80 when attempting to call Galaxy at 'https://galaxy.ansible.com/api/':
86.80 'CustomHTTPSConnection' object has no attribute 'cert_file'.
86.80 'CustomHTTPSConnection' object has no attribute 'cert_file'
86.80 ERROR! Unknown error when attempting to call Galaxy at 'https://galaxy.ansible.com/api/': 'CustomHTTPSConnection' object has no attribute 'cert_file'. 'CustomHTTPSConnection' object has no attribute 'cert_file'
------
Dockerfile:22
--------------------
  21 |     COPY requirements.txt requirements.yml ansible.cfg bootstrap /teuthology/
  22 | >>> RUN \
  23 | >>>     cd /teuthology && \
  24 | >>>     mkdir ../archive_dir && \
  25 | >>>     mkdir log && \
  26 | >>>     chmod +x /teuthology/bootstrap && \
  27 | >>>     PIP_INSTALL_FLAGS="-r requirements.txt" ./bootstrap
  28 |     COPY . /teuthology
--------------------
ERROR: failed to solve: process "/bin/sh -c cd /teuthology &&     mkdir ../archive_dir &&     mkdir log &&     chmod +x /teuthology/bootstrap &&     PIP_INSTALL_FLAGS=\"-r requirements.txt\" ./bootstrap" did not complete successfully: exit code: 1
Service 'teuthology' failed to build : Build failed