Created automation for Packages.Microsoft publishing #431

ellbosch · 2020-03-04T09:05:13Z

In the spirt of our deb and rpm packaging, I have created a Dockerized approach to automate* the publishing of deb and rpm packages to the Packages.Microsoft repository. Automation is needed to support the very long list of Linux distributions we've previously supported. A future PR will provide improved testing for our supported Linux distributions.

* The Repo CLI used to publish packages requires manual installation. This solution is not perfectly automated due to this complexity. Therefore, a step-by-step guide is provided.

Details of this PR:

Dockerfile which creates container for publishing. Packages.Microsoft provides a Repo CLI API for publishing packages, which requires an AMD64 environment. We use a Dockerfile to create the staging environment needed for Repo CLI installation and script automation.
publish.sh script uploads to either testing or prod. When called in the pre-built Docker container, this script will publish a deb or rpm to each supported package in our json files. It requires an argument for specifying either 'testing' or 'prod', and will not execute unless specified.
JSON files for testing and prod-supported repos. The objects refers to identifiers known by Packages.Microsoft, which are pulled during publish.sh execution. Each entity represents a Linux distribution where we've previously published mssql-cli, or its future versions we've yet to support.
Step-by-step docs. A README is provided to get the Docker container up and running, to install the Repo CLI, and to execute the publish.sh script.
Folder organization changes. A new 'release_scripts' folder has been added to the repo root, where automated tasks for publishing may reside (currently hosts a 'Packages.Microsoft' folder, only).

pensivebrian · 2020-03-07T21:53:47Z

release_scripts/Packages.Microsoft/Dockerfile

+# Download latest stable deb and rpm packages
+WORKDIR /root
+RUN wget https://mssqlcli.blob.core.windows.net/daily/deb/mssql-cli-dev-latest.deb
+RUN wget https://mssqlcli.blob.core.windows.net/daily/rpm/mssql-cli-dev-latest.rpm


Let's move away from the 'latest' package and use explicit version numbers. This issue is, a later build might have overwritten the latest and we would know. If we reference the explicit version, this is less likely to happen.

pensivebrian · 2020-03-07T21:55:37Z

release_scripts/Packages.Microsoft/supported_repos_testing.json

@@ -0,0 +1,280 @@
+[


Is this coming from a package? If so, can we just download package and avoid checking this is? My concern is these values can change.

These values won't change—they map to repos in Packages.Microsoft. If you query the Repo CLI you can get this information.

The key bit of data is the repository ID, which needs to be in the root config when uploading a package. Without some sort of automation, you'd need to look up the repository ID for each Linux repo you're uploading to, and edit the root config manually.

pensivebrian · 2020-03-07T21:55:56Z

release_scripts/Packages.Microsoft/supported_repos_prod.json

@@ -0,0 +1,126 @@
+


same deal here - do we have to check this in?

See my comment above. We'd need to store this data somewhere for automation to work. There isn't sensitive data so I thought a JSON would be ok.

As an alternative solution, we could store these values in a database.

remove json, keep urls and add to script

ellbosch · 2020-04-14T19:07:31Z

release_scripts/Packages.Microsoft/publish.sh

+
+local_repo=$1
+config_file=/root/.repoclient/config.json
+deb_pkg=/root/mssql-cli-dev-latest.deb


remove dev-latest

ellbosch · 2020-04-14T19:14:23Z

release_scripts/Packages.Microsoft/publish.sh

+    # with config updated, publish deb or rpm package
+    if [ $repo_type == "apt" ]; then
+        echo "Publishing .deb for $repo_url..."
+        repoclient package add $deb_pkg


add to list of commands in readme

pensivebrian

I'm wondering if this script should generate a .sh with all reproclient add commands. Then, one would launch the .sh. This way, one can review up front what commands will be run, and then decide them individually, are just run the script.

pensivebrian · 2020-04-15T15:57:16Z

release_scripts/Packages.Microsoft/publish.sh

+# specificies destination testing repos for publishing
+repo_url_testing=( \
+    "microsoft-ubuntu-trusty-prod" \
+    "microsoft-ubuntu-xenial-prod" \


aren't some of these prod? I see some of these exists in both lists.

Nope—the URL naming is unfortunately confusing and inconsistent at times.

In some cases, the URL name is the same for both the prod and testing distribution. However, repo ID's are always unique, and we query for the desired repo ID by filtering for URL name and distribution type (i.e. prod or testing).

pensivebrian · 2020-04-15T16:00:52Z

release_scripts/Packages.Microsoft/publish.sh

+    # -r specifies the destination repository (by ID)
+    if [ $repo_type == "apt" ]; then
+        echo "Publishing .deb for $repo_url..."
+        repoclient package add $deb_pkg -r $repo_id


what if there is an error? If there is an error, I think we should abort.

Great minds think alike!!! I had problems terminating the script during this step so I just made a change :)

pensivebrian · 2020-04-15T16:05:07Z

release_scripts/Packages.Microsoft/Dockerfile

@@ -0,0 +1,31 @@
+FROM amd64/ubuntu:18.04 AS builder


You can mentioned there are issues with this. Does this work out of the box?

The only issue I remember bringing up was the requirement for using amd64 to install the Repo Client. So with Docker, this should work out of the box.

I thought you mention stdio was involved for a license agreement or something.

Oh, yes, STDIO is required for responding to a prompt with the repo client installer. That's why the repo client installation is out of the dockerfile's scope—instead, the dockerfile completes requirements needed for repo client installation.

I see. Can we have a similar script for those who don't use docker?

If you don't use Docker, then it's just a matter of building a Linux machine that can install the repo client. If that's done, then the only remaining steps are:

Install Repo Client

Call the publish.sh file

No other scripts are needed.

pensivebrian · 2020-04-15T16:07:14Z

release_scripts/Packages.Microsoft/README.md

+```sh
+release_scripts/Packages.Microsoft/publish.sh $(pwd) 'prod'
+```
+


We should add a note about what todo if there is an error or need to refresh the binaries with a new version. Can we just re-run the tool with the same package versions?

Do you mean package versions of mssql-cli? Yes.

ops, wrong thread. Cool, we should doc this since many repo's (nuget, pypi) don't support this (or didn't in the recent past)

Let me run this back with the Repo Client team to confirm.

pensivebrian · 2020-04-15T16:08:23Z

release_scripts/Packages.Microsoft/publish.sh

@@ -0,0 +1,100 @@
+# Publishes deb and rpm packages to Packages.Microsoft repo


I'm curious why we didn't do this in python?

In hindsight... we could have. Pros and cons here. I thought the bash script would have been more elegant because I pictured fewer complications than our result.

What's nice here is you don't need any Python, it works in any container with bash so this script can theoretically be used in any Linux container (using arm). But Python would have been a cleaner looking script.

ellbosch requested review from pensivebrian and udeeshagautam March 4, 2020 09:05

pensivebrian reviewed Mar 7, 2020

View reviewed changes

pensivebrian requested changes Mar 7, 2020

View reviewed changes

ellbosch requested a review from pensivebrian April 2, 2020 22:10

ellbosch force-pushed the ellbosch/msftrepo branch 2 times, most recently from c88eff9 to 6ec21e7 Compare April 11, 2020 02:34

ellbosch commented Apr 14, 2020

View reviewed changes

ellbosch force-pushed the ellbosch/msftrepo branch 2 times, most recently from 80f09e1 to 89292a7 Compare April 15, 2020 15:51

pensivebrian requested changes Apr 15, 2020

View reviewed changes

pensivebrian reviewed Apr 15, 2020

View reviewed changes

ellbosch and others added 14 commits April 29, 2020 19:58

Builds container; starting README work and automation

42566bd

Shell script edits config based on json

66ed28a

fixed repo cli requirements

622b7d9

automated publishing task

513c5af

Added support for testing/prod param

d17272f

documentation for publish.sh

754eef1

updated comment in gitignore

7f5409a

edited sections needed for completion

d430bc5

updated list of supported and unsupported repos

c248abe

added debian 8 and 9 support

a8ff81f

added nano dependency

b69f54f

package links point to specific version

b5bb396

updated docs

4cfce5a

support for packages with version numbers

c86b632

ellbosch added 10 commits April 29, 2020 19:58

updated docs

c528c93

updated docs

33355f6

specificies repo id on CL call

bc0bede

exit loop if something failed with command

b5ff7c0

added config with blank values

e804ee1

adds --print option

0d3de29

moved wget to publish.sh

2287280

updated docs

6916ad5

changed repo list to url-distro key value

b061542

changed --print to --upload

69ae0d7

ellbosch force-pushed the ellbosch/msftrepo branch from 7a226dc to 69ae0d7 Compare April 30, 2020 02:58

added rpm urls

148dcee

pensivebrian approved these changes Apr 30, 2020

View reviewed changes

removed trusty; fixed rpm file name

21a4d6c

ellbosch merged commit 30dcde1 into master Apr 30, 2020

ellbosch deleted the ellbosch/msftrepo branch April 30, 2020 21:15

		@@ -0,0 +1,100 @@
		# Publishes deb and rpm packages to Packages.Microsoft repo

Created automation for Packages.Microsoft publishing #431

Created automation for Packages.Microsoft publishing #431

Uh oh!

Conversation

ellbosch commented Mar 4, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pensivebrian left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ellbosch Apr 15, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ellbosch Apr 15, 2020 •

edited

Loading