Skip to content

Commit

Permalink
Merge pull request #8817 from AdamBrousseau/multi_artifactory
Browse files Browse the repository at this point in the history 
Support uploading to multiple Artifactory servers
  • Loading branch information
@pshipton
pshipton committed Mar 27, 2020
2 parents 386b29b + 7caada9 commit bf841bb
Show file tree
Hide file tree
Showing 4 changed files with 321 additions and 42 deletions.
105 changes: 102 additions & 3 deletions buildenv/jenkins/ARTIFACTORY.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
<!--
Copyright (c) 2019, 2019 IBM Corp. and others
Copyright (c) 2019, 2020 IBM Corp. and others
This program and the accompanying materials are made available under
the terms of the Eclipse Public License 2.0 which accompanies this
Expand Down Expand Up @@ -47,16 +47,25 @@ Edit the file in /home/jenkins/artifactory/etc/binarystore.xml and create the fo
</config>
```

Edit the Artifactory defaults values `/home/jenkins/artifactory/bin/artifactory.default`
```
export ARTIFACTORY_HOME=/home/jenkins/artifactory/artifactory-oss-6.5.2
export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-ppc64el
```
Also for the UNB setup, Artifactory was complaining that `-Xss` should be set to greater than default 256k. I set it to 512k.

# Reverse Proxy
Next is to install a reverse proxy. In this case, NGINX was installed
A Reverse Proxy was used so that https can be configured.

To install NGINX use this command
` sudo apt install nginx`

Then the reverse configuration needs to be set
Then the reverse configuration needs to be set.
The template for the reverse proxy was taken from https://www.jfrog.com/confluence/display/RTF/Configuring+NGINX and there were some changes to it.
This is the copy that is currently being used

`/etc/nginx/sites-enabled/artifactory.conf`
```
## server configuration
server {
Expand Down Expand Up @@ -111,6 +120,48 @@ All of the lines with the `# managed by Certbot` were added when adding in the s

Those lines do not need to be added as they were automatically added by certbot

`/etc/nginx/sites-available/artifactory.conf`
```
## server configuration
server {
listen 443 ssl;
listen 80 ;
server_name _;
if ($http_x_forwarded_proto = '') {
set $http_x_forwarded_proto $scheme;
}
## add ssl entries when https has been set in config
ssl_certificate /etc/nginx/ssl/artifactory.crt;
ssl_certificate_key /etc/nginx/ssl/artifactory.key;
ssl_session_cache shared:SSL:1m;
ssl_prefer_server_ciphers on;
## Application specific logs
## access_log /var/log/nginx/artifactory.log timing;
## error_log /var/log/nginx/artifactory-error.log;
rewrite ^/$ /artifactory/webapp/ redirect;
rewrite ^/artifactory/?(/webapp)?$ /artifactory/webapp/ redirect;
chunked_transfer_encoding on;
client_max_body_size 0;
location / {
proxy_read_timeout 900;
proxy_pass_header Server;
proxy_cookie_path ~*^/.* /;
if ( $request_uri ~ ^/artifactory/(.*)$ ) {
proxy_pass http://localhost:8081/artifactory/$1;
}
proxy_pass http://localhost:8081/artifactory/;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header Host $http_host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
}
```

I am getting my certificate from Let's Encrypt and their install tool certbot. Firstly, I installed certbot:
```
sudo apt-get update
Expand All @@ -122,10 +173,27 @@ sudo apt-get install certbot python-certbot-nginx
```
Then I used the certbot automated script to add the certificates and to automatically renew the certificates:
```
sudo certbot -nginx
sudo certbot --nginx
```
Follow the onscreen instructions and everything should work out.

UNB uses a self-signed certificate. Because it is behind a VPN, Let's Encrypt can't auth the cert.

Create the keys
https://www.digitalocean.com/community/tutorials/how-to-create-a-self-signed-ssl-certificate-for-apache-in-ubuntu-18-04
```
openssl req -x509 -nodes -days 99999 -newkey rsa:2048 -keyout /etc/ssl/private/apache-selfsigned.key -out /etc/ssl/certs/apache-selfsigned.crt
```
Update the 2 files above with the key and crt generated from the above command.

Useful nginx commands
```
service nginx start
```
```
service nginx reload
```

# Open the Ports
Log into OpenStack (https://openpower-controller.osuosl.org/auth/login/?next=/project/) and go to network security groups
Click on `http` (if it is not created, create the rule)
Expand Down Expand Up @@ -162,6 +230,8 @@ Give the permission a name like `Jenkins-perm` and include the `ci-eclipse-openj

Under the Admin tab on the left side, select Services -> Backups and make sure that they are disabled. To see if they are disabled, click on the backup and see if the enabled button is checked. If it is, uncheck it and save. The backups take up too much memory for what ends up being needed.

Under the Admin tab on the left side, select Configuration -> General Configuration. Disable trash can. Save.

Now logoff of the admin account and log into the Jenkins account. Edit the profile of Jenkins by clicking on Welcome, Jenkins -> Edit Profile. Insert the password to unlock the account and generate the API key. Copy that key to the secrets repo and update Jenkins so that it can upload artifacts.

I also added in a separate shell command to restart Artifactory.
Expand All @@ -179,6 +249,35 @@ This line was added
@reboot bash /home/jenkins/artifactory/artifactory-oss-6.5.2/bin/artifactory_restart.sh
```

# Configure UNB nodes for curl redirection

Create a curl wrapper to redirect OSU requests to UNB. Also strip off user info as the creds will be wrong for the UNB server. Take note of what is default curl before you override.
```
/usr/local/bin/curl
```
```
echo $@ | sed 's#--user .*:.* ##' | xargs /usr/bin/curl --resolve 140-211-168-230-openstack.osuosl.org:443:192.168.10.216
```

# Configure UNB nodes to accept self-signed certificate (upload only)

UNB nodes uploading to UNB Artifactory need to trust the self signed cert. Curl doesn't need it for download since we use the -k option to ignore the cert.
https://confluence.atlassian.com/kb/how-to-import-a-public-ssl-certificate-into-a-jvm-867025849.html

```
openssl s_client -connect 192.168.10.216:443 -servername ub18p8art.casa.cs.unb.ca:443 < /dev/null | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > public.crt
```
Substitute whatever JAVA_HOME the node agent is running.
```
/usr/lib/jvm/java-8-openjdk-amd64/bin/keytool -import -alias ub18p8art.casa.cs.unb.ca -keystore /usr/lib/jvm/java-8-openjdk-amd64/jre/lib/security/cacerts -file public.crt
```
```
Enter keystore password:changeit
```
```
Trust this certificate? [no]: y
```

# Useful Links

https://www.jfrog.com/confluence/display/RTF/Installing+on+Linux+Solaris+or+Mac+OS#InstallingonLinuxSolarisorMacOS-ManualInstallation
Expand Down
96 changes: 75 additions & 21 deletions buildenv/jenkins/common/build.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -325,45 +325,48 @@ def archive_sdk() {
}
}
}
if (ARTIFACTORY_SERVER) {
if (ARTIFACTORY_CONFIG) {
def specs = []
def sdkSpec = ["pattern": "${OPENJDK_CLONE_DIR}/${SDK_FILENAME}",
"target": "${ARTIFACTORY_UPLOAD_DIR}",
"target": "${ARTIFACTORY_CONFIG['uploadDir']}",
"props": "build.buildIdentifier=${BUILD_IDENTIFIER}"]
specs.add(sdkSpec)
def testSpec = ["pattern": "${OPENJDK_CLONE_DIR}/${TEST_FILENAME}",
"target": "${ARTIFACTORY_UPLOAD_DIR}",
"target": "${ARTIFACTORY_CONFIG['uploadDir']}",
"props": "build.buildIdentifier=${BUILD_IDENTIFIER}"]
specs.add(testSpec)
def debugImageSpec = ["pattern": "${OPENJDK_CLONE_DIR}/${DEBUG_IMAGE_FILENAME}",
"target": "${ARTIFACTORY_UPLOAD_DIR}",
"target": "${ARTIFACTORY_CONFIG['uploadDir']}",
"props": "build.buildIdentifier=${BUILD_IDENTIFIER}"]
specs.add(debugImageSpec)
if (params.ARCHIVE_JAVADOC) {
def javadocSpec = ["pattern": "${OPENJDK_CLONE_DIR}/${JAVADOC_FILENAME}",
"target": "${ARTIFACTORY_UPLOAD_DIR}",
"target": "${ARTIFACTORY_CONFIG['uploadDir']}",
"props": "build.buildIdentifier=${BUILD_IDENTIFIER}"]
specs.add(javadocSpec)
}
def uploadFiles = [files : specs]
def uploadSpec = JsonOutput.toJson(uploadFiles)
upload_artifactory(uploadSpec)
env.CUSTOMIZED_SDK_URL = "${ARTIFACTORY_URL}/${ARTIFACTORY_UPLOAD_DIR}${SDK_FILENAME}"

// Always use the default server link for the description and for test.
env.CUSTOMIZED_SDK_URL = "${ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']]['url']}/${ARTIFACTORY_CONFIG['uploadDir']}${SDK_FILENAME}"
currentBuild.description += "<br><a href=${CUSTOMIZED_SDK_URL}>${SDK_FILENAME}</a>"

if (fileExists("${TEST_FILENAME}")) {
TEST_LIB_URL = "${ARTIFACTORY_URL}/${ARTIFACTORY_UPLOAD_DIR}${TEST_FILENAME}"
env.CUSTOMIZED_SDK_URL += " " + TEST_LIB_URL
TEST_LIB_URL = "${ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']]['url']}/${ARTIFACTORY_CONFIG['uploadDir']}${TEST_FILENAME}"
currentBuild.description += "<br><a href=${TEST_LIB_URL}>${TEST_FILENAME}</a>"
env.CUSTOMIZED_SDK_URL += " " + TEST_LIB_URL
}
if (fileExists("${DEBUG_IMAGE_FILENAME}")) {
DEBUG_IMAGE_LIB_URL = "${ARTIFACTORY_URL}/${ARTIFACTORY_UPLOAD_DIR}${DEBUG_IMAGE_FILENAME}"
DEBUG_IMAGE_LIB_URL = "${ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']]['url']}/${ARTIFACTORY_CONFIG['uploadDir']}/${DEBUG_IMAGE_FILENAME}"
currentBuild.description += "<br><a href=${DEBUG_IMAGE_LIB_URL}>${DEBUG_IMAGE_FILENAME}</a>"
}
if (params.ARCHIVE_JAVADOC) {
if (fileExists("${JAVADOC_FILENAME}")) {
JAVADOC_LIB_URL = "${ARTIFACTORY_URL}/${ARTIFACTORY_UPLOAD_DIR}${JAVADOC_FILENAME}"
env.CUSTOMIZED_SDK_URL += " " + JAVADOC_LIB_URL
JAVADOC_LIB_URL = "${ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']]['url']}/${ARTIFACTORY_CONFIG['uploadDir']}${JAVADOC_FILENAME}"
currentBuild.description += "<br><a href=${JAVADOC_LIB_URL}>${JAVADOC_FILENAME}</a>"
echo "Javadoc:'${JAVADOC_LIB_URL}"
}
}
echo "CUSTOMIZED_SDK_URL:'${CUSTOMIZED_SDK_URL}'"
Expand Down Expand Up @@ -432,44 +435,92 @@ def archive_diagnostics() {
} else {
sh "find . -name 'core.*.dmp' -o -name 'javacore.*.txt' -o -name 'Snap.*.trc' -o -name 'jitdump.*.dmp' | sed 's#^./##' | tar -zcvf ${DIAGNOSTICS_FILENAME} -T -"
}
if (ARTIFACTORY_SERVER) {
if (ARTIFACTORY_CONFIG) {
def uploadSpec = """{
"files":[
{
"pattern": "${DIAGNOSTICS_FILENAME}",
"target": "${ARTIFACTORY_UPLOAD_DIR}",
"target": "${ARTIFACTORY_CONFIG['uploadDir']}",
"props": "build.buildIdentifier=${BUILD_IDENTIFIER}"
}
]
}"""
upload_artifactory(uploadSpec)
DIAGNOSTICS_FILE_URL = "${ARTIFACTORY_URL}/${ARTIFACTORY_UPLOAD_DIR}${DIAGNOSTICS_FILENAME}"
upload_artifactory_core(ARTIFACTORY_CONFIG['defaultGeo'], uploadSpec)
DIAGNOSTICS_FILE_URL = "${ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']]['url']}/${ARTIFACTORY_CONFIG['uploadDir']}${DIAGNOSTICS_FILENAME}"
currentBuild.description += "<br><a href=${DIAGNOSTICS_FILE_URL}>${DIAGNOSTICS_FILENAME}</a>"
} else {
archiveArtifacts artifacts: "${DIAGNOSTICS_FILENAME}", fingerprint: false
}
}

def upload_artifactory(uploadSpec) {
def server = Artifactory.server ARTIFACTORY_SERVER
// Loop all the servers and upload if we've determined we should do so
for (geo in ARTIFACTORY_CONFIG['geos']) {
if (ARTIFACTORY_CONFIG[geo]['uploadBool']) {
/*
* If the server is behind a vpn and we aren't on a node behind the vpn,
* save the sdk and upload it later on a machine behind the vpn
*/
if (ARTIFACTORY_CONFIG[geo]['vpn'] == 'true' && !NODE_LABELS.contains("ci.geo.${geo}")) {
if (!ARTIFACTORY_CONFIG['stashed']) {
// Stash only what test needs (CUSTOMIZED_SDK_URL)
stash includes: "**/${SDK_FILENAME},**/${TEST_FILENAME}", name: 'sdk'
ARTIFACTORY_CONFIG['stashed'] = true
ARTIFACTORY_CONFIG['uploadSpec'] = uploadSpec
}
ARTIFACTORY_CONFIG[geo]['uploaded'] = false
} else {
upload_artifactory_core(geo, uploadSpec)
ARTIFACTORY_CONFIG[geo]['uploaded'] = true
}
}
}
}

def upload_artifactory_core(geo, uploadSpec) {
echo "Uploading to '${geo}'..."
def server = Artifactory.server ARTIFACTORY_CONFIG[geo]['server']
// set connection timeout to 10 mins to avoid timeout on slow platforms
server.connection.timeout = 600

def buildInfo = Artifactory.newBuildInfo()
buildInfo.retention maxBuilds: ARTIFACTORY_NUM_ARTIFACTS, maxDays: ARTIFACTORY_DAYS_TO_KEEP_ARTIFACTS, deleteBuildArtifacts: true
buildInfo.retention maxBuilds: ARTIFACTORY_CONFIG[geo]['numArtifacts'], maxDays: ARTIFACTORY_CONFIG[geo]['daysToKeepArtifacts'], deleteBuildArtifacts: true
// Add BUILD_IDENTIFIER to the buildInfo. The UploadSpec adds it to the Artifact info
buildInfo.env.filter.addInclude("BUILD_IDENTIFIER")
buildInfo.env.capture = true

//Retry uploading to Artifactory if errors occur
pipelineFunctions.retry_and_delay({
server.upload spec: uploadSpec, buildInfo: buildInfo;
server.publishBuildInfo buildInfo},
if (!ARTIFACTORY_CONFIG[geo]['vpn']){server.publishBuildInfo buildInfo}},
3, 300)

// Write URL to env so that we can pull it from the upstream pipeline job
env.ARTIFACTORY_URL = server.getUrl()
env.ARTIFACTORY_CREDS = server.getCredentialsId()
ARTIFACTORY_CONFIG[geo]['url'] = server.getUrl()
ARTIFACTORY_CONFIG[geo]['creds'] = server.getCredentialsId()
// If this is the default server, save the creds to pass to test
if (geo == ARTIFACTORY_CONFIG['defaultGeo']) {
env.ARTIFACTORY_CREDS = ARTIFACTORY_CONFIG[ARTIFACTORY_CONFIG['defaultGeo']].creds
}
}

def upload_artifactory_post() {
// Determine if we didn't do any Artifactory uploads because of vpn.
// At the time of writing this code, we would only hit this case if we compiled at OSU on plinux and needed to upload to UNB.
if (ARTIFACTORY_CONFIG && ARTIFACTORY_CONFIG['stashed']) {
for (geo in ARTIFACTORY_CONFIG['geos']) {
if (ARTIFACTORY_CONFIG[geo]['uploadBool'] && !ARTIFACTORY_CONFIG[geo]['uploaded']) {
// Grab a node with the same geo as the server so we can upload
node("ci.geo.${geo}") {
try {
unstash 'sdk'
upload_artifactory_core(geo, ARTIFACTORY_CONFIG['uploadSpec'])
} finally {
cleanWs()
}
}
}
}
}
}

def add_node_to_description() {
Expand All @@ -486,6 +537,8 @@ def build_all() {
// Cleanup in case an old build left anything behind
cleanWs()
add_node_to_description()
// Setup Artifactory now that we are on a node. This determines which server(s) we push to.
variableFile.set_artifactory_config()
get_source()
build()
archive_sdk()
Expand All @@ -495,6 +548,7 @@ def build_all() {
}
}
}
upload_artifactory_post()
}
}
}
Expand Down
Loading

0 comments on commit bf841bb

Please sign in to comment.