Deployment of build.opensuse.org

Saray Cabrera Padrón edited this page Nov 20, 2018 · 68 revisions

Running the deployment

  1. Check get the commit id of the currently deployed code

      osc api /about
    
  2. Verify that newly built packages are available on the OBS instance by checking the versions:

      zypper search --details obs-api
    

    Get the commit id of the currently available (published) packages

    NOTE: If new packages don't get published due to running builds, you might want to disable the GitHub / OBS integration.

  3. Inspect the code (diff) for migrations and other changes that require attention. You can do this in GitHub with the following url:

    https://github.com/openSUSE/open-build-service/compare/commit1...commit2
    

    i.e using curl curl -L "https://github.com/openSUSE/open-build-service/compare/80cb3011e...3c1e92ea8.diff"

    Go to the "When there is no migration" section if there are not migrations, or to the "When there are migrations" section otherwise.

When there is no migration

  1. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again.

  2. Update the packages

    zypper update --details obs-api
    
  3. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

  4. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

When there are migrations

  1. Wait for our maintenance window

    Thursday, 8am to 10am CET / CEST

  2. Check /etc/motd, it lists all pull requests that manually got applied to the deployment. Either merge them and wait for new packages or make sure you manually apply the changes after deployment again.

  3. Put apache into maintenance mode in /etc/sysconfig/apache2

      APACHE_SERVER_FLAGS="STATUS MAINTENANCE"
    
  4. Restart apache

      rcapache2 restart
    
  5. Update the packages

      zypper update --details obs-api
    
  6. Migrate the database

      run_in_api rails db:migrate:with_data
    
  7. Put apache out of maintenance mode in /etc/sysconfig/apache2

      APACHE_SERVER_FLAGS="STATUS"
    
  8. Restart apache and delayed jobs

      rcapache2 restart
    
  9. Check the state of OBS. Especially the areas that have been touched by code you just deployed. Make sure delayed jobs are running.

  10. Apply monkey patches if there are any and modify the Monkey Patch section of /etc/motd accordingly.

  11. Check that the Factory dashboard still works https://build.opensuse.org/project/dashboard/openSUSE:Factory. If not, inform Ludwig / DimStar / Coolo.

Merging Pull Requests With Migrations

Pull Requests which introduce migrations (which are expected to raise a downtime) are required to get merged on Wednesday morning (Those pull requests are labeled as Merge on Wednesday) to make sure the migrations can be run in the maintenance window.

The creator of the pull request is responsible to merge those changes.

Troubleshooting

Delayed Jobs

For delayed jobs, clockworkd and indexing sphinx there is a systemd target:

systemctl obs-api-support.target start|stop|status|restart

To make sure everything is running fine:

# The systemd target should display as active (running), in green
systemctl status obs-api-support.target
# All dependant services should display active (running), in green
systemctl list-dependencies obs-api-support.target

If one of this is already running and systemctl obs-api-support.target stop does not stop it, you have to manually kill it (ps aux | grep delayed) otherwise restarting will fail.

To get an overview about remaining jobs, you can run:

run_in_api rails runner script/delayed_job_stats.rb

Search

Sometimes, we search in OBS for a package/project name and we don't obtain any results, but we know the package/project exist (we know the direct link to it).

Both projects and packages are indexed for searchs. There is a rake task that updates the indexes periodically. However, sometimes the task doesn't run. To check this and re-run, please follow the steps:

  • Go to production server via SSH.
  • $ run_in_api rails ts:index
  • You might have the following output
WARNING: The following indexing guard files exist:  /srv/www/obs/api/db/sphinx/production/ts---all.tmp
These files indicate indexing is already happening. If that is not the case, these files should be deleted to ensure all indices can be processed
  • As we can read in the output, the task is not going to run again if the guard file is there. Maybe the process didn't finish the last time it run or there was a deployment at the same time and it got stuck.
  • Then check if that guard file is too old. ls -lah /srv/www/obs/api/db/sphinx/production/ts---all.tmp
  • If the file is new (less than a few hours), please wait, maybe the ts:index task is running now. Stop reading here and check again after one hour.
  • If the file is old, remove it.
  • Run $ run_in_api rails ts:index again. It will take a few minutes.
  • Check in OBS WebUI if search works now.

Disabling the rpm test suite

In case that the deployment caused some breakage and you might need to build some new packages quickly. In that case you might want to temporarily disable the test suite in our rpm spec. To do that add %disable_obs_test_suite 1 to the project config of OBS:Server:Unstable. Important: It needs to go into the Macros: ... :Macros section.

Downgrading packages

It might happen that a deployment breaks OBS badly and you need to get it to work again quickly. In that case you can check if zypper still has the old packages in it's cache. Just run zypper se --details and verify that the package version you want is still available. If that's the case, run zypper in --oldpackage obs-api-$VERSION.$ARCH to downgrade the package.

Disabling OBS integration in GitHub

OBS packages are build whenever a PR get's merged to master. This might delay publishing of built packages. To prevent this, disable the OBS integration in GitHub:

  1. Go to settings of the OBS GitHub project
  2. Select the 'Integration & services' tab and click on 'Edit' in the OBS column
  3. Uncheck the 'Activate' checkbox and 'Update services'
  4. Once the deployment is done activate the checkbox again ;-)

Deployment of the obs-cloud-uploader package

  • ssh to our cloud uploader instance
  • Run zypper up obs-cloud-uploader
Clone this wiki locally
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.