Zenhub Enterprise 3.5.3

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the "Important Upgrade Instructions for Administrators" section.

IMPORTANT: Zenhub Enterprise 3.5.3 supports GitHub Enterprise versions: 3.6, 3.7, 3.8

What's new in Zenhub Enterprise 3.5.3

Features

• No new features in this patch release

Security Fixes

• Package security updates

Bug Fixes

• Fixed several links in the Account management "API" page that linked to the Zenhub Cloud developer site instead of the self-hosted developer site

Changes

• Support bundle generation on ZHE for VM is more resilient
• Added support bundle generation for ZHE for k8s
• Added the ability to directly upload support bundles to Zenhub Enterprise Support
• Optimized the application to make fewer API calls to GitHub
• Removed numerous warning level logs

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• Open source repos search is not functional.

VM Embedded Component Versions

Component	Version
Ubuntu	22.04.2
K3s	v1.22.10+k3s1
Kubernetes	v1.22.10
Kustomize	v4.5.3
Fluentd	v1.15-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.17.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.4 -> ZHE 3.5, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.4 for any reason.

• You must be running ZHE 3.4.X, 3.5.0, 3.5.1, or 3.5.2 to perform the upgrade to ZHE 3.5.3.

• For users currently running Zenhub Enterprise, look for an email from our team which will include a download link for the 3.5.3 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The minimum version of Kubernetes required to run ZHE 3.5 is v1.22

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• Two new ingress definitions have been added to this release: options/ingress/ingress-api-public.yaml and options/ingress/ingress-devsite.yaml. You will need to set the TLS secret name and annotations to suit your ingress, similar to your other configured Zenhub ingress objects.
• ZHE 3.5 adds a developer site that contains documentation and a graphiql explorer for the new GraphQL API. To configure the subdomain for this site, you must set a value for the new configuration key graphiql_explorer_subdomain in the kustomization.yaml file. The value can be left as the default developers if desired. The resulting site will be available at developers-<subdomain_suffix>.<domain_tld>. You will need to setup a new DNS route for this domain pointing to the same location as your main DNS route for Zenhub.
• ZHE 3.5 adds PgBouncer in front of your postgres database. PgBouncer runs in the cluster. To make set up more simple, your configuration will need to be updated so that postgres_url now instead points to the PgBouncer service, and the new pgbouncer_url key should be configured to point to the actual postgres database. The kustomization.yaml file for ZHE 3.5 contains these instructions as comments and configuration placeholders, however feel free to reach out to enterprise@zenhub.com for support if you have any questions, since this change is not trivial.
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Run configmap-generator.sh

Once you have setup all the configurations in your copy of kustomization.yaml and are ready to deploy to your cluster, you must run the configmap-generator.sh script that will fill placeholders in our Kubernetes manifests with your configuration values.

To run the script, navigate to the root of the directory containing your kustomization.yaml file, which also contains the configmap-generator.sh script. Then, run the script with these two commands:

chmod 700 configmap-generator.sh
./configmap-generator.sh

4. Application updates

In your new kustomization.yaml, confirm that the cluster image tags are set to the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.5.3
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.5.3
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.5.3
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.5.3
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.5.3
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.5.3
  - name: devsite
    newName: us.gcr.io/zenhub-public/devsite
    newTag: zhe-3.5.3
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• First, delete any raptor-db-migrate and sanitycheck jobs so they may be recreated without errors:

Make sure the status of the jobs are Complete and not Running

kubectl -n <your_dedicated_namespace> delete job/raptor-db-migrate job/sanitycheck

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

5. Database changes

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 3.4.

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

6. Re Deploy Application

⚠️ NOTE: Running these commands is only required if you are upgrading from ZHE 3.4.

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

7. Finalize

• Securely store the updated kustomization.yaml

Zenhub Enterprise 3.5.2

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the "Important Upgrade Instructions for Administrators" section.

IMPORTANT: Zenhub Enterprise 3.5.2 supports GitHub Enterprise versions: 3.6, 3.7, 3.8

What's new in Zenhub Enterprise 3.5.2

Features

• No new features in this patch release

Security Fixes

• Package security updates

Bug Fixes

• Fixed bug causing issues to occasionally disappear from the board when the board is filtered
• Link to update Github profile in the 'Account management' menu has been updated to point to the respective Github host instead of github.com
• Fixed bug causing the zhe-config --support-bundle not to work in previous ZHE 3.5 releases
• Fixed bug causing the zhe-config --import-images not to work in previous ZHE 3.5 releases

Changes

• Added support for GHE 3.8
• The default domain name for the new developer site has been updated to https://developers-<subdomain>.<domain>
• TLS is now configurable for the new developer site on ZHE for VM

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• Open source repos search is not functional.

VM Embedded Component Versions

Component	Version
Ubuntu	22.04.2
K3s	v1.22.10+k3s1
Kubernetes	v1.22.10
Kustomize	v4.5.3
Fluentd	v1.15-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.17.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.4 -> ZHE 3.5, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.4 for any reason.

• You must be running ZHE 3.4.X, 3.5.0, or 3.5.1 to perform the upgrade to ZHE 3.5.2.

• For users currently running Zenhub Enterprise, look for an email from our team which will include a download link for the 3.5.2 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The minimum version of Kubernetes required to run ZHE 3.5 is v1.22

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• Two new ingress definitions have been added to this release: options/ingress/ingress-api-public.yaml and options/ingress/ingress-devsite.yaml. You will need to set the TLS secret name and annotations to suit your ingress, similar to your other configured Zenhub ingress objects.
• ZHE 3.5 adds a developer site that contains documentation and a graphiql explorer for the new GraphQL API. To configure the subdomain for this site, you must set a value for the new configuration key graphiql_explorer_subdomain in the kustomization.yaml file. The value can be left as the default developers if desired. The resulting site will be available at developers-<subdomain_suffix>.<domain_tld>. You will need to setup a new DNS route for this domain pointing to the same location as your main DNS route for Zenhub.
• ZHE 3.5 adds PgBouncer in front of your postgres database. PgBouncer runs in the cluster. To make set up more simple, your configuration will need to be updated so that postgres_url now instead points to the PgBouncer service, and the new pgbouncer_url key should be configured to point to the actual postgres database. The kustomization.yaml file for ZHE 3.5 contains these instructions as comments and configuration placeholders, however feel free to reach out to enterprise@zenhub.com for support if you have any questions, since this change is not trivial.
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Run configmap-generator.sh

Once you have setup all the configurations in your copy of kustomization.yaml and are ready to deploy to your cluster, you must run the configmap-generator.sh script that will fill placeholders in our Kubernetes manifests with your configuration values.

To run the script, navigate to the root of the directory containing your kustomization.yaml file, which also contains the configmap-generator.sh script. Then, run the script with these two commands:

chmod 700 configmap-generator.sh
./configmap-generator.sh

4. Application updates

In your new kustomization.yaml, confirm that the cluster image tags are set to the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.5.2
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.5.2
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.5.2
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.5.2
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.5.2
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.5.2
  - name: devsite
    newName: us.gcr.io/zenhub-public/devsite
    newTag: zhe-3.5.2
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• First, delete any raptor-db-migrate and sanitycheck jobs so they may be recreated without errors:

Make sure the status of the jobs are Complete and not Running

kubectl -n <your_dedicated_namespace> delete job/raptor-db-migrate job/sanitycheck

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

5. Database changes

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 3.4.

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

6. Re Deploy Application

⚠️ NOTE: Running these commands is only required if you are upgrading from ZHE 3.4.

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

7. Finalize

• Securely store the updated kustomization.yaml

Zenhub Enterprise 3.5.1

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the "Important Upgrade Instructions for Administrators" section.

IMPORTANT: Zenhub Enterprise 3.5.1 supports GitHub Enterprise versions: 3.6, 3.7

What's new in Zenhub Enterprise 3.5.1

Features

• No new features in this patch release

Security Fixes

• Package security updates

Bug Fixes

• Fixed a UI issue affecting navigation on the Zenhub Enterprise Account Dashboard.
• Fixed an issue preventing publishing the Zenhub Firefox extension.

Changes

• No new changes in this patch release

Known Issues

• On ZHE VM, the new developer site and GraphQL explorer use an unsecured connection.
• Generating support bundles is not working.
• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• Open source repos search is not functional.

VM Embedded Component Versions

Component	Version
Ubuntu	22.04.2
K3s	v1.22.10+k3s1
Kubernetes	v1.22.10
Kustomize	v4.5.3
Fluentd	v1.15-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.17.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.4 -> ZHE 3.5, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.4 for any reason.

• You must be running ZHE 3.4.X or 3.5.0 to perform the upgrade to ZHE 3.5.1.

• For users currently running Zenhub Enterprise, look for an email from our team which will include a download link for the 3.5.1 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The minimum version of Kubernetes required to run ZHE 3.5 is v1.22

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• Two new ingress definitions have been added to this release: options/ingress/ingress-api-public.yaml and options/ingress/ingress-devsite.yaml. You will need to set the TLS secret name and annotations to suit your ingress, similar to your other configured Zenhub ingress objects.
• ZHE 3.5 adds a developer site that contains documentation and a graphiql explorer for the new GraphQL API. To configure the subdomain for this site, you must set a value for the new configuration key graphiql_explorer_subdomain in the kustomization.yaml file. The value can be left as the default developers if desired. The resulting site will be available at developers.<subdomain_suffix>.<domain_tld>.
• ZHE 3.5 adds PgBouncer in front of your postgres database. PgBouncer runs in the cluster. To make set up more simple, your configuration will need to be updated so that postgres_url now instead points to the PgBouncer service, and the new pgbouncer_url key should be configured to point to the actual postgres database. The kustomization.yaml file for ZHE 3.5 contains these instructions as comments and configuration placeholders, however feel free to reach out to enterprise@zenhub.com for support if you have any questions, since this change is not trivial.
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Run configmap-generator.sh

Once you have setup all the configurations in your copy of kustomization.yaml and are ready to deploy to your cluster, you must run the configmap-generator.sh script that will fill placeholders in our Kubernetes manifests with your configuration values.

To run the script, navigate to the root of the directory containing your kustomization.yaml file, which also contains the configmap-generator.sh script. Then, run the script with these two commands:

chmod 700 configmap-generator.sh
./configmap-generator.sh

4. Application updates

In your new kustomization.yaml, update the cluster image tags to use the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.5.1
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.5.1
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.5.1
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.5.1
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.5.1
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.5.1
  - name: devsite
    newName: us.gcr.io/zenhub-public/devsite
    newTag: zhe-3.5.1
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• First, delete the raptor-db-migrate and sanitycheck jobs so they may be recreated without errors:

Make sure the status of the jobs are Complete and not Running

kubectl -n <your_dedicated_namespace> delete job/raptor-db-migrate job/sanitycheck

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

5. Database changes

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 3.4.

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

6. Re Deploy Application

⚠️ NOTE: Running these commands is only required if you are upgrading from ZHE 3.4.

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

7. Finalize

• Securely store the updated kustomization.yaml

Zenhub Enterprise 3.5.0

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: Zenhub Enterprise 3.5.0 supports GitHub Enterprise versions: 3.6, 3.7

What's new in Zenhub Enterprise 3.5.0

Features

• New public GraphQL API with configurable rate limits
• New self hosted developer site available at developers.<your-zenhub-hostname>
  
• Legacy REST API rate limits now configurable
• New system health check tool added to VM
• Added PG Bouncer to ZHE for K8s

Security Fixes

• Package security updates

Bug Fixes

• Fixed bug with Board 2.0 where changes to an issue's metadata would occasionally cause the ticket to disappear on the board
• Old ZHE image versions are now cleaned up during upgrades
• Added K3s internal certificate rotation to VM upgrades
• PgBouncer configuration updated to perform better at scale
• Nginx Gateway configuration updated to perform better at scale
• Fixed air-gapped installation method for new and updated python packages

Changes

• Added back support for load balancer TLS termination which was removed in ZHE 3.4
• Base image updated to Ubuntu 22.04
• Kustomize updated to 4.5
• Kubernetes updated to 1.22.10
• PgBouncer updated to 1.17
• Fluentd updated to 1.15

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• Open source repos search is not functional.
• The Zenhub Firefox extension fails validation and cannot be published, only the Chrome extension can be published as is.

VM Embedded Component Versions

Component	Version
Ubuntu	22.04.2
K3s	v1.22.10+k3s1
Kubernetes	v1.22.10
Kustomize	v4.5.3
Fluentd	v1.15-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.17.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.4 -> ZHE 3.5, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.4 for any reason.

• You must be running ZHE 3.4.X to perform the upgrade to ZHE 3.5.

• For users currently running Zenhub Enterprise, look for an email from our team which will include a download link for the 3.5.0 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The minimum version of Kubernetes required to run ZHE 3.5 is v1.22

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• Two new ingress definitions have been added to this release: options/ingress/ingress-api-public.yaml and options/ingress/ingress-devsite.yaml. You will need to set the TLS secret name and annotations to suit your ingress, similar to your other configured Zenhub ingress objects.
• ZHE 3.5 adds a developer site that contains documentation and a graphiql explorer for the new GraphQL API. To configure the subdomain for this site, you must set a value for the new configuration key graphiql_explorer_subdomain in the kustomization.yaml file. The value can be left as the default developers if desired. The resulting site will be available at developers.<subdomain_suffix>.<domain_tld>.
• ZHE 3.5 adds PgBouncer in front of your postgres database. PgBouncer runs in the cluster. To make set up more simple, your configuration will need to be updated so that postgres_url now instead points to the PgBouncer service, and the new pgbouncer_url key should be configured to point to the actual postgres database. The kustomization.yaml file for ZHE 3.5 contains these instructions as comments and configuration placeholders, however feel free to reach out to enterprise@zenhub.com for support if you have any questions, since this change is not trivial.
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Run configmap-generator.sh

Once you have setup all the configurations in your copy of kustomization.yaml and are ready to deploy to your cluster, you must run the configmap-generator.sh script that will fill placeholders in our Kubernetes manifests with your configuration values.

To run the script, navigate to the root of the directory containing your kustomization.yaml file, which also contains the configmap-generator.sh script. Then, run the script with these two commands:

chmod 700 configmap-generator.sh
./configmap-generator.sh

4. Application updates

In your new kustomization.yaml, update the cluster image tags to use the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.5.0
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.5.0
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.5.0
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.5.0
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.5.0
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.5.0
  - name: devsite
    newName: us.gcr.io/zenhub-public/devsite
    newTag: zhe-3.5.0
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

5. Database changes

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 3.4.

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

6. Re Deploy Application

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

7. Finalize

• Securely store the updated kustomization.yaml

Zenhub Enterprise 3.4.1

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: Zenhub Enterprise 3.4.1 supports GitHub Enterprise versions: 3.4, 3.5, 3.6

What's new in Zenhub Enterprise 3.4.1

Features

• No new features in this patch release

Security Fixes

• Package security updates
• OS security updates
• Updated supported cipher suites

Bug Fixes

• Fixed a bug causing database migration jobs to fail if a deleted repository had contained an Epic that was present on a Roadmap.

Changes

• Added back support for a layer 4/7 load balancer performing TLS termination in front of the application that was inadvertently removed in v3.4.0

Known Issues

• Occasionally in Board 2.0, updates to an issue's metadata can cause the issue to disappear on the board. Refreshing the board displays the issue again.
• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.

VM Embedded Component Versions

Component	Version
Ubuntu	20.04.4
K3s	v1.21.11+k3s1
Kubernetes	v1.21.11
Kustomize	v3.9.4
Fluentd	v1.6-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.15.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.3 -> ZHE 3.4, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.3 for any reason.

• You must be running ZHE 3.3.X or 3.4.0 to perform the upgrade to ZHE 3.4.1.

• For users currently running Zenhub Enterprise, look for an email from our team which will include a download link for the 3.4.1 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: Amazon DocumentDB 3.6 can be used for the ZHE 3.3->3.4 upgrade if not using TLS with DocumentDB. Otherwise, DocumentDB 3.6 must be upgraded to 4.0 before the ZHE 3.3->3.4 upgrade. Please see the AWS documentation and reach out to support@zenhub.com if you have any questions.

1. Prepare

• Get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• A new ingress definition has been added to this release in options/ingress/ingress-cable.yaml. You will need to set the TLS secret name and annotations to suite your ingress, similar to your other configured Zenhub ingress objects.
• If you are using your own registry and not Zenhub's registry, upload the new images tagged with this release to your registry.

3. Application updates

In your new kustomization.yaml, confirm that the cluster image tags are set to the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of Zenhub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.4.1
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.4.1
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.4.1
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.4.1
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.4.1
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.4.1
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• First, delete any raptor-db-migrate and sanitycheck jobs so they may be recreated without errors:

Make sure the status of the jobs are Complete and not Running

kubectl -n <your_dedicated_namespace> delete job/raptor-db-migrate job/sanitycheck

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

4. Database changes

⚠️ NOTE: Running this script is only required if you are upgrading from ZHE 3.3.

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

5. Redeploy the application

⚠️ NOTE: Running these commands is only required if you are upgrading from ZHE 3.3

The data migration script run in the previous step will have scaled down the application in order to safely migrate data without active database writes occurring. Now that the migration is complete, the application needs to be deployed to the normal desired operational state:

• First perform a diff to check what will change (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct and there are no errors, you can deploy to the cluster via:

kustomize build . | kubectl apply -f-

6. Finalize

• Securely store the updated kustomization.yaml

Zenhub Enterprise 3.4.0

These release notes are for Zenhub Enterprise for Virtual Machine and Zenhub Enterprise for Kubernetes.

• For all users using Zenhub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: Zenhub Enterprise 3.4.0 supports GitHub Enterprise versions: 3.4, 3.5

What's new in Zenhub Enterprise 3.4.0

Features

Epic flyover on the roadmap

You will no longer be tabbed out into a new page once you click on your epic on the roadmap. Instead we've enabled the flyover similar to what we do for issues on the board so you never have to leave your roadmap when reading or updating your epics.

When creating epics, auto add to roadmap with the ability to opt out

Adding epics to a roadmap can be time consuming so we do it for you if you want us to. Whenever you create an epic we'll add it to your workspace roadmap by checking a box at the bottom of the epic create page. You can opt out if you don't want to add your epic by unchecking this box. If you quick create an epic we'll automatically add it to the roadmap for you. You can always remove your epics from your roadmap if you don't want them there as you normally could.

Productivity Insights out of beta

Spend less time pouring over reports and see exactly what you need, directly on the board. These insights will:

Automatically break down work in progress into development & review time so you can understand how long each phase is taking.
Let you know when review time is taking significantly longer than development time so you’re aware of blockers, sooner.

Additionally, we’ve added anomalies tracking to show you which issues took abnormally long to complete. This will help you identify why cycle time might be rising, spot trends among particularly challenging issues, and prompt conversation among the team.

Roadmap drag and drop

You can now manually reorder items on the Roadmap using drag and drop! Sort your Roadmap by start date, end date, or manually by using the drop-down menu in the top right corner.

Board 2.0

Our Board 2.0 has been built for speed and efficiency from the ground up. We’ve observed the following improvements:

In addition to our speed improvements, we’ve also taken the opportunity to add a few new features to the board and make incremental improvements to existing ones. We’ve made the following feature changes to the board:

Easily groom your board with an improved drag and drop experience

We’ve rebuilt our drag and drop experience for the board to make moving and re-ordering issues on the board buttery smooth. We’ve also added new subtle animations and made tweaks behind the scenes to improve reliability when dragging issues across the board.

Identify bottlenecks with separated pipeline counts

Our pipeline counts provide a quick summary of issues and PR’s in a pipeline. To help our users better understand this count, we’ve broken out the count into separate categories for Issues, Pull Requests, and Points. This change trades the obscurity of the previous count for the accuracy and detail of seeing card-specific counts at a glance.

Easily scan your board with less clutter thanks to a cleaner board view

Over time, we’ve heard feedback from users that the board can appear cluttered once each issue has several labels, sprints, and releases that are tied to each card. To help create a cleaner board view, we’ve elongated each issue card and removed some of the auxiliary information from each card in the board view.

Minimize interruptions with increased reliability

As part of our board rebuild, we’ve taken the time to address several bugs and reliability issues that were occasionally present in the old board. By addressing these issues we have created a more reliable board experience that results in less error for users when taking board actions, including creating issues.

Multi-select can now be used within the Closed pipeline to take bulk actions on closed tickets, including:

• Applying labels
• Moving tickets out of the Closed pipeline
• Assigning epics

Condensed view - a new way to view your board

Abstracts the majority of ticket information, removes spacing between tickets and increases the number of tickets viewable at a given time on the board.

You can read more about our new board in this blog post!

Please note: for the first version, drag & drop is enabled for top-level items, meaning you can reorder Projects, and Epics that aren't inside a Project.

Security Fixes

• Package security updates
• Updated supported cipher suites
• TLS 1.2 is now an minimum version for ZHE for VM
• Closed vulnerability that allowed attackers to discover URLs of uploaded files and images

Bug Fixes

• Embedded version of runc has been updated, resolving the bug with containerd logs filling up
• If a user accesses Zenhub and is not part of any repos, the button prompting them to create a repository no longer incorrectly redirects to github.com

Changes

• Added a new deployment to better manage sidekiq queues and improve live application updates
• Secondary, redundant swap has been removed
• Fixed scaling logic to handle atypical VM sizes (ex. 8x16)
• Mongo upgrade from 4.4.9 to 4.4.13 on ZHE for VM
• K3s upgraded to 1.21.11 on ZHE for VM
• K3s upgrades are now air-gapped and enforced
• Initial Zenhub air-gapped installation now working properly
• Chrome Extension updated to Manifest V3

Known Issues

• Occasionally in Board 2.0, updates to an issue's metadata can cause the issue to disappear on the board. Refreshing the board displays the issue again.
• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.

VM Embedded Component Versions

Component	Version
Ubuntu	20.04.4
K3s	v1.21.11+k3s1
Kubernetes	v1.21.11
Kustomize	v3.9.4
Fluentd	v1.6-debian-1
Postgresql	11.16
MongoDB	4.4.13
RabbitMQ	3.8.31
Redis	6.2.7
PgBouncer	1.15.0

Important Upgrade Instructions for Administrators

Zenhub Enterprise for Virtual Machine

⚠️ NOTE: For the upgrade from ZHE 3.3 -> ZHE 3.4, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.3 for any reason.

• You must be running ZHE 3.3 to perform the upgrade to ZHE 3.4.

• For users currently running Zenhub Enterprise 3.3, look for an email from our team which will include a download link for the 3.4.0 upgrade package. The upgrade process is detailed in our documentation.

Zenhub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: Amazon DocumentDB 3.6 can be used for the ZHE 3.3->3.4 upgrade if not using TLS with DocumentDB. Otherwise, DocumentDB 3.6 must be upgraded to 4.0 before the ZHE 3.3->3.4 upgrade. Please see the AWS documentation and reach out to support@zenhub.com if you have any questions.

1. Prepare

• You need to get the kustomization.yaml you configured to setup Zenhub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing conf...

Zenhub Enterprise 3.1.3

ZenHub Enterprise 3.1.3

These release notes are for ZenHub Enterprise (On-Premise) Virtual Machine with GitHub Enterprise Server.

• For users currently running ZenHub Enterprise 2.44, you may migrate directly to 3.1.3 instead of an earlier 3.1 release.
• For all users using ZenHub on github.com, please check out our new feature announcements on our product changelog.

IMPORTANT: ZenHub Enterprise 3.1.3 supports the following GitHub Enterprise versions: 2.22, and 3.0

What's new in ZenHub Enterprise 3.1.3 since ZenHub Enterprise 3.1.2

Security Fixes

Bug Fixes

• Fixed issue with incompatible openshift package version that could cause problems with data backups, restores, migrations, and upgrades.
• Fixed air-gapped installation method.
• Removed secondary swap file.

Changes

• Increased the minimum disk requirement to 90GB.

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround has been added to the documentation.
• Environments migrated from ZHE 2.44 that have not used the same CRYPTO_PASS will see a blank page in the Milestones view on the web app.
• File uploads in the web application (not extension) are limited to 1MB in size.

That's all for now! Stay tuned for more updates coming soon! 🎉

ZenHub Enterprise 3.3.3

These release notes are for ZenHub Enterprise for Virtual Machine and ZenHub Enterprise for Kubernetes.

• For all users using ZenHub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: ZenHub Enterprise 3.3.3 supports GitHub Enterprise versions: 3.3, 3.4

What's new in ZenHub Enterprise 3.3.3

Features

• No new features in this patch release.

Security Fixes

• Package security updates.

Bug Fixes

• Fixed Firefox extension installation for Firefox v100 and greater.

Changes

• No changes in this patch release.

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• The version of runc embedded in the K3s installation contains a logging bug which can potentially fill up containerd logs.
• 'Manage organizations' dashboard does not have dark-mode theming.
• The alignment of the license management options in the 'Manage organizations' dashboard is too close to the radio buttons.
• Partial username search does not return any users in the Team members section of the 'Manage organizations' dashboard, only an exact search works.
• When using the browser extension and logged into GitHub Enterprise as an administrator, the black enterprise version banner at the top of the page sits over ZenHub elements.
• If a user accesses ZenHub and is not part of any repos, the button prompting them to create a repository redirects to GitHub Cloud.

VM Embedded Component Versions

Component	Version
Ubuntu	20.04.4
K3s	v1.20.13+k3s1
Kubernetes	v1.20.13
Kustomize	v3.9.4
Fluentd	v1.6-debian-1
Postgresql	11.14
MongoDB	4.4.9
RabbitMQ	3.8
Redis	6.2
PgBouncer	1.15.0

Important Upgrade Instructions for Administrators

ZenHub Enterprise for Virtual Machine

⚠️ NOTE: Specifically for the upgrade from ZHE 3.2 -> ZHE 3.3, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.2 for any reason.

• You must be running either ZHE 3.2 or 3.3 to perform the upgrade to ZHE 3.3.3.

• For users currently running ZenHub Enterprise 3.2 or 3.3, look for an email from our team which will include a download link for the 3.3.3 upgrade package. The upgrade process is detailed in our documentation.

ZenHub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The upgrade from ZHE 3.2 -> ZHE 3.3 will require a service outage, due to the underlying data migration being performed. Make sure to back up your data and test your backups prior to performing the upgrade. For large datasets allocate an appropriate amount of time for the migration.

⚠️ NOTE: Amazon DocumentDB 3.6 can be used for the ZHE 3.2->3.3 upgrade if not using TLS with DocumentDB. Otherwise, DocumentDB 3.6 must be upgraded to 4.0 before the ZHE 3.2->3.3 upgrade. Please see the AWS documentation and reach out to support@zenhub.com if you have any questions.

1. Prepare

• You need to get the kustomization.yaml you configured to setup ZenHub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• If you are using your own registry and not ZenHub's registry, upload the new images tagged with this release to your registry.

3. Database changes

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

4. Application updates

In your new kustomization.yaml, update the cluster image tags to use the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of ZenHub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.3.3
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.3.3
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.3.3
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.3.3
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.3.3
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.3.3
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct, you can deploy the cluster via:

kustomize build . | kubectl apply -f-

5. Finalize

• Securely store the updated kustomization.yaml

ZenHub Enterprise 3.3.2

These release notes are for ZenHub Enterprise for Virtual Machine and ZenHub Enterprise for Kubernetes.

• For all users using ZenHub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: ZenHub Enterprise 3.3.2 supports GitHub Enterprise versions: 3.3, 3.4

What's new in ZenHub Enterprise 3.3.2

Features

• No new features in this patch release.

Security Fixes

• Updated to Ubuntu 20.04.4 LTS.
• Package security updates.

Bug Fixes

• Fixed package version upgrade that could cause issues with upgrades, data backups, and data restores.
• Fixed file downloads on ZHE for K8s.
• Fixed release report data migration for upgrades from 3.2 to 3.3

Changes

• Improved logrotation of application logs for ZHE for VM.
• Updated websocket routing configuration for ZHE for K8s deployments, allowing for better performing horizontal application scaling in high load environments.
• Enhanced ZHE version compatibility check for ZHE for VM upgrades.

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• The version of runc embedded in the K3s installation contains a logging bug which can potentially fill up containerd logs.
• 'Manage organizations' dashboard does not have dark-mode theming.
• The alignment of the license management options in the 'Manage organizations' dashboard is too close to the radio buttons.
• Partial username search does not return any users in the Team members section of the 'Manage organizations' dashboard, only an exact search works.
• When using the browser extension and logged into GitHub Enterprise as an administrator, the black enterprise version banner at the top of the page sits over ZenHub elements.
• If a user accesses ZenHub and is not part of any repos, the button prompting them to create a repository redirects to GitHub Cloud.

VM Embedded Component Versions

Component	Version
Ubuntu	20.04.4
K3s	v1.20.13+k3s1
Kubernetes	v1.20.13
Kustomize	v3.9.4
Fluentd	v1.6-debian-1
Postgresql	11.14
MongoDB	4.4.9
RabbitMQ	3.8
Redis	6.2
PgBouncer	1.15.0

Important Upgrade Instructions for Administrators

ZenHub Enterprise for Virtual Machine

⚠️ NOTE: Specifically for the upgrade from ZHE 3.2 -> ZHE 3.3, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.2 for any reason.

• You must be running either ZHE 3.2.0, 3.2.1, or 3.3.0 to perform the upgrade to ZHE 3.3.2.

• For users currently running ZenHub Enterprise 3.2 or 3.3.0, look for an email from our team which will include a download link for the 3.3.2 upgrade package. The upgrade process is detailed in our documentation.

ZenHub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The upgrade from ZHE 3.2 -> ZHE 3.3 will require a service outage, due to the underlying data migration being performed. Make sure to back up your data and test your backups prior to performing the upgrade. For large datasets allocate an appropriate amount of time for the migration.

⚠️ NOTE: Amazon DocumentDB 3.6 can be used for the ZHE 3.2->3.3 upgrade if not using TLS with DocumentDB. Otherwise, DocumentDB 3.6 must be upgraded to 4.0 before the ZHE 3.2->3.3 upgrade. Please see the AWS documentation and reach out to support@zenhub.com if you have any questions.

1. Prepare

• You need to get the kustomization.yaml you configured to setup ZenHub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
• Populate the new kustomization.yaml with your existing configuration values

Be sure to replace any additional customized configuration (such as ingress, TLS configuration) with your original configuration from before as well.

• If you are using your own registry and not ZenHub's registry, upload the new images tagged with this release to your registry.

3. Database changes

• Run the script found in k8s-cluster/update/zhe-upgrade.sh via the commands below:

If you use the ZenHub registry

cd update
./zhe-upgrade.sh yourNamespace

If you use your own registry

cd update
./zhe-upgrade.sh yourNamespace yourRegistryName

4. Application updates

In your new kustomization.yaml, update the cluster image tags to use the new images for this release:

If you are using your own registry, ensure that the newName fields are configured for that instead of ZenHub's

images:
  - name: kraken-webapp
    newName: us.gcr.io/zenhub-public/kraken-webapp
    newTag: zhe-3.3.2
  - name: kraken-extension
    newName: us.gcr.io/zenhub-public/kraken-extension
    newTag: zhe-3.3.2
  - name: kraken-zhe-admin
    newName: us.gcr.io/zenhub-public/kraken-zhe-admin
    newTag: zhe-3.3.2
  - name: raptor-backend
    newName: us.gcr.io/zenhub-public/raptor-backend
    newTag: zhe-3.3.2
  - name: toad-backend
    newName: us.gcr.io/zenhub-public/toad-backend
    newTag: zhe-3.3.2
  - name: sanitycheck
    newName: us.gcr.io/zenhub-public/sanitycheck
    newTag: zhe-3.3.2
  - name: busybox
    newName: docker.io/library/busybox
    newTag: latest
  - name: nginx
    newName: docker.io/library/nginx
    newTag: latest

• Then perform a diff to check what the upgrade will do (this command must be run from the directory that contains your kustomization.yaml

kustomize build . | kubectl diff -f-

• If everything looks correct, you can deploy the cluster via:

kustomize build . | kubectl apply -f-

5. Finalize

• Securely store the updated kustomization.yaml

ZenHub Enterprise 3.3.0

These release notes are for ZenHub Enterprise for Virtual Machine and ZenHub Enterprise for Kubernetes.

• For all users using ZenHub on github.com, please check out our new feature announcements on our product changelog.
• For administrators planning upgrades, please refer to the section "Important Upgrade Instructions for Administrators".

IMPORTANT: ZenHub Enterprise 3.3 supports GitHub Enterprise versions: 3.2, 3.3

What's new in ZenHub Enterprise 3.3.0

Features

Planning poker

Team members can now estimate issues individually and record a history of who estimated what. No more guessing who was part of the process or where the estimate came from.

We've added the ability to optionally alert a teammate that they've been added to planning poker for an issue. This feature automatically posts an issue comment tagging a team mate which notifies them that an estimate is needed.

You can also request estimates from your whole team at once via Planning Poker Groups! When requesting estimates during a round of Planning Poker, users will be grouped based on your previous selections. This means you can choose your entire team in just one click rather than selecting them one-by-one.

Quick issue creation

A new [+] button has been added to the top of each of each of your pipelines to quickly create issues within the respective pipeline. Click, type, submit. It's as easy as that!

To quickly create an epic, click the new [+] button on any pipeline and then click on the issue icon to see it transform into an epic.

Remove epic start and end dates

You can now remove the start and end dates for an epic directly from the date selection widget.

Export and share roadmaps

You can now export roadmaps to an SVG file via the Share button.

GitHub Projects Importer

For those who want to break away from GitHub Projects and come over to ZenHub, you now have an easy way to import your project and issues over with just a couple of clicks.

Sprint Insights Beta

Right from your Board you’ll be able to: see ZenHub Sprint progress, understand changes in scope, see average velocity and compare to last period, add unfinished Issues to your next Sprint, and more.

A Toast to Completed Sprints

We've added a Sprint Completed toast that provides a short summary of your team's progress over the last sprint. Use this as a way to glean sprint information, update your team on progress or as a reminder to celebrate a completed sprint.

Dark mode has arrived in ZenHub!

Dark mode lovers attest to its ability to reduce eye strain in low light conditions. Most importantly, we think it happens to look especially snazzy! Available on both the ZenHub web app and the ZenHub browser extension for GitHub.

Enabling dark mode can be done in just one simple step, no matter where you use ZenHub!

• For those using the browser extension in GitHub, ZenHub’s dark mode will automatically update based on your GitHub preferences. In other words, if you use dark mode in GitHub, you’ll have dark mode in ZenHub!
• For those that use the web app, dark mode can be conveniently activated by clicking the up arrow on your profile on the bottom left, and toggling on dark mode. That’s it, you’ve got dark mode!

Security Fixes

• Updated to Ubuntu 20.04.3 LTS.
• Updated embedded Kubernetes in ZHE for VM.
• Updated embedded backend databases in ZHE for VM.
• Added TLS support for external MongoDB in ZHE for K8s.

Bug Fixes

• File uploads in the web application are no longer limited to 1MB in size.
• A page refresh is no longer needed to view ZenHub properties when opening a GitHub issue with the ZenHub browser extension installed with newer versions of GitHub Enterprise.
• Dragging PRs with long names on the Board no longer clips the previous issue.
• Two byte characters in input fields no longer prematurely entered.
• Increased pod liveness check start delay for ZenHub backend K8s pods, resolving rough pod startup.

Changes

• Increased default minimum disk size from 80GB to 90GB, increasing root filesystem from 20GB to 30GB for new installations.
• Fine tuned log rotation to decrease log sizes in ZHE for VM.
• Adding default backend to ingress.
• Optimized memory allocation for small deployments.
• Added zhe-config --update-tls command to enable updating TLS certificates without a full application restart.
• Added zhe-config --update-license command for updating your ZenHub license without bringing down the application.

Known Issues

• Too many authentication failures may be reported when attempting to SSH into the VM, which can be caused by multiple SSH keys loaded in the connecting workstation's SSH agent. A workaround is present in the documentation.
• The version of runc embedded in the K3s installation contains a logging bug which can potentially fill up containerd logs.
• 'Manage organizations' dashboard does not have dark-mode theming.
• The alignment of the license management options in the 'Manage organizations' dashboard is too close to the radio buttons.
• Partial username search does not return any users in the Team members section of the 'Manage organizations' dashboard, only an exact search works.
• When using the browser extension and logged into GitHub Enterprise as an administrator, the black enterprise version banner at the top of the page sits over ZenHub elements.
• If a user accesses ZenHub and is not part of any repos, the button prompting them to create a repository redirects to GitHub Cloud.

VM Embedded Component Versions

Component	Version
Ubuntu	20.04.3
K3s	v1.20.13+k3s1
Kubernetes	v1.20.13
Kustomize	v3.9.4
Fluentd	v1.6-debian-1
Postgresql	11.14
MongoDB	4.4.9
RabbitMQ	3.8
Redis	6.2
PgBouncer	1.15.0

Important Upgrade Instructions for Administrators

ZenHub Enterprise for Virtual Machine

⚠️ NOTE: Specifically for the upgrade from ZHE 3.2 -> ZHE 3.3, the regular rollback steps cannot be used. Please perform a machine level snapshot prior to upgrading to use as a backup in case you need to revert back to ZHE 3.2 for any reason.

• You must be running either ZHE 3.2.0 or ZHE 3.2.1 to perform the upgrade to ZHE 3.3.0.

• For users currently running ZenHub Enterprise 3.2.0 or 3.2.1, look for an email from our team which will include a download link for the 3.3.0 upgrade package. The upgrade process is detailed in our documentation.

ZenHub Enterprise for Kubernetes

• Some extra upgrade instructions have been added specifically for this release due to some database changes that accompany the application updates. Please follow the upgrade instructions below for this release:

⚠️ NOTE: The upgrade from ZHE 3.2 -> ZHE 3.3 will require a service outage, due to the underlying data migration being performed. Make sure to back up your data and test your backups prior to performing the upgrade. For large datasets allocate an appropriate amount of time for the migration.

⚠️ NOTE: Amazon DocumentDB 3.6 can be used for the ZHE 3.2->3.3 upgrade if not using TLS with DocumentDB. Otherwise, DocumentDB 3.6 must be upgraded to 4.0 before the ZHE 3.2->3.3 upgrade. Please see the AWS documentation and reach out to support@zenhub.com if you have any questions.

1. Prepare

• You need to get the kustomization.yaml you configured to setup ZenHub
• Perform a diff to make sure no outstanding changes are waiting to be applied

kustomize build . | kubectl diff -f-

It should exit 0 and only display a warning of unused variables. If changes are pending, apply them before starting the upgrade process.

• Make a copy of your existing kustomization.yaml and keep it handy for the next step

2. Update kustomization.yaml

• Check out the zenhub-enterprise repository at the tag of this release
  ...