Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs/self-host: add k8s requirements & other fixes #2222

Merged
merged 5 commits into from
Oct 14, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion contents/docs/self-host/deploy/aws.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,17 @@ tags:
- aws
---

import ClusterRequirementsSnippet from './snippets/cluster-requirements'
import InstallingSnippet from './snippets/installing'
import UpgradingSnippet from './snippets/upgrading'
import UninstallingSnippet from './snippets/uninstalling'
import TryUnsecureSnippet from './snippets/tryunsecure'

First, we need to set up a Kubernetes Cluster, see [Setup EKS - eksctl](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html). Follow the "Managed nodes - Linux" guide. The default nodes (2x m5.large) work well for running PostHog.
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
First, we need to set up a Kubernetes cluster (see the official AWS [documentation](https://docs.aws.amazon.com/eks/latest/userguide/getting-started-eksctl.html) for more info).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are missing the part about how to get the expandable volume. This should be documented to make users life easier (& we should test that we can make this work).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we shouldn't follow the route to copy and paste volume type specific how-to-steps from 3rd party documentations as they'll be likely out of date the second after we merge this PR.

Those are the volume types currently supported:

  • gcePersistentDisk
  • awsElasticBlockStore
  • Cinder
  • glusterfs
  • rbd
  • Azure File
  • Azure Disk
  • Portworx
  • FlexVolume
  • CSI

should we document and keep up to date the procedures on how to enable the setting for all of them?

I think it’s important to note that for PostHog we recommend the use of storage classes with expandable volumes, but then it’s up to our users to decide if and how they want to implement that.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the same spirit as the suggestion for users to use DO. We want to avoid making it seem complicated to get PostHog up and running, if someone knows what they are doing they will do it anyway.

So in that spirit I do think it's worth it for us to document this explicitly for each platform, but we don't have to copy their documentation, we can just link to the right place similarly as we did for cluster creation. That yes will get out of date potentially and a user will ask about it in users slack & then we'll update it.

Btw one of our goals is to minimize the amount of time it takes for someone to spin up & maintain their self-hosted instance (we don't have metrics for this yet defined, but <10min average install time ; <15min average monthly maintenance time seem like good goals).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This sounds like it could be a separate PR to outline the Volumes we support with documentation.
The only Volume types I would support here are

  • gcePersistentDisk
  • awsElasticBlockStore
  • Azure Disk

Anything beyond these and the user is using a stack that is going to be pretty custom.

Let's land this PR and add a todo to document these. No reason to hold up shipping this.


<ClusterRequirementsSnippet />

## Chart configuration

Here's the minimal required `values.yaml` that we'll be using later. You can find an overview of the parameters that can be configured during installation under [chart configuration](/docs/self-host/deploy/configuration).
```yaml
Expand Down
11 changes: 9 additions & 2 deletions contents/docs/self-host/deploy/azure.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,21 @@ title: Deploying to Azure
sidebarTitle: Azure
sidebar: Docs
showTitle: true
tags:
- azure
---

import ClusterRequirementsSnippet from './snippets/cluster-requirements'
import InstallingSnippet from './snippets/installing'
import UpgradingSnippet from './snippets/upgrading'
import UninstallingSnippet from './snippets/uninstalling'
import TryUnsecureSnippet from './snippets/tryunsecure'

First, we need to set up a Kubernetes Cluster, see [Creation with Azure portal](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal#create-an-aks-cluster). Make sure your cluster has enough resources to run PostHog (total minimum 4 vcpu & 8GiB RAM).
First, we need to set up a Kubernetes cluster (see the official Azure [documentation](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal#create-an-aks-cluster) for more info).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same how can we get a qualifying cluster, is following the instructions good enough or not. cc @fuziontech to setup Azure account for PostHog team

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's your suggestion on this? Should we add more documentation on top of the official Azure documentation?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My suggestion very specifically is for every platform specific guide:

  1. Test what the default is,
  2. Add to the docs a sentence along the lines of "At the time of writing by default <uses/doesn't use> expanding volumes"
  3. if not the default figure out how to do it & expand on how to do it.

Additionally in the "other platforms" I'd call out that this is something they'd want to check explicitly as many platforms default is non-expandable.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's simply state that for all platforms outside DigitalOcean you must ensure that Kubernetes supports expandable volumes. We only have so many resources to spend on documentation here and the only platform I think we can all agree that should be as turn key as possible is DigitalOcean. The rest you are going to need to have some sort of understanding of what is going on in the K8s infrastructure or you will have a bad time.

Options for deploying PostHog sorted by how K8s familiar you are:

  1. Cloud
  2. DigitalOcean
    -----Should have baseline understanding of K8s beyond here-----
  3. AWS/GCP/Azure
  4. Rest of World


<ClusterRequirementsSnippet />

## Chart configuration

Here's the minimal required `values.yaml` that we'll be using later. You can find an overview of the parameters that can be configured during installation under [chart configuration](/docs/self-host/deploy/configuration).
```yaml
Expand All @@ -30,7 +37,7 @@ kafka:

<InstallingSnippet />


### Lookup external IP

```console
Expand Down
110 changes: 44 additions & 66 deletions contents/docs/self-host/deploy/digital-ocean.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -3,89 +3,55 @@ title: Deploying to Digital Ocean
sidebarTitle: Digital Ocean
sidebar: Docs
showTitle: true
tags:
- do
- digitalocean
---

import DOIPAddressSnippet from './snippets/do-ip-address'
import GetUrlSnippet from './snippets/get-url'
import GetIPAddressSnippet from './snippets/get-ip-address'
import PostInstallSnippet from './snippets/post-install'
import ClusterRequirementsSnippet from './snippets/cluster-requirements'
import InstallingSnippet from './snippets/installing'
import UpgradingSnippet from './snippets/upgrading'
import UninstallingSnippet from './snippets/uninstalling'
import TryUnsecureSnippet from './snippets/tryunsecure'

## Why Digital Ocean
[Digital Ocean](https://digitalocean.com) is one of the most well-established Cloud Providers. Compared to AWS, where the amount of options and configuration can be overwhelming, Digital Ocean is generally simpler to use and faster to get running.

[Digital Ocean](https://digitalocean.com) is one of the most well-established Cloud Providers. Compared to AWS, where the amount of options and configuration can be overwhelming, Digital Ocean is generally simpler to use and faster to get running.
<br />
The first thing you'll need is a get a Digital Ocean account. You can click on the badge below to get US$100 in credit over 60 days (i.e. run PostHog for free for ~2 months).

You can then either follow the [1-click install](#1-click-install) or the [manual install](#manual-install).
tiina303 marked this conversation as resolved.
Show resolved Hide resolved

The first thing you'll need is a get a Digital Ocean account. You can click on the badge below to get $100 in credit over 60 days (i.e. run PostHog for free for 2 months).
<br />

<center>
<a href="https://www.digitalocean.com/?refcode=6a26a2c395b0&utm_campaign=Referral_Invite&utm_medium=Referral_Program&utm_source=badge" title="DigitalOcean Referral">
<img alt="DigitalOcean Referral Badge" src="https://web-platforms.sfo2.cdn.digitaloceanspaces.com/WWW/Badge%201.svg" />
</a>
</center>
<br />

You can then either follow the [1-click install](#1-click-install) or the [manual install](#manual-install).
<br />

## 1-click install
There is a [1-click install](https://marketplace.digitalocean.com/apps/posthog-1) option to deploy PostHog via the [DigitalOcean Marketplace](https://marketplace.digitalocean.com/).
The DigitalOcean UI will ask you if you want to install PostHog on an already provisioned Kubernetes cluster or if you want to create a new one.

### Marketplace UI
There is a [1-click option to deploy PostHog](https://marketplace.digitalocean.com/apps/posthog-1) on DigitalOcean Marketplace UI.

If you don't already have a cluster setup you should follow the getting started guide to setup kubeconfig. The minimal suggested cluster capacity is using **two of the smallest production nodes**.

You will also need to secure your PostHog deployment.

### CLI

Alternatively use [doctl](https://github.com/digitalocean/doctl) which also sets up kubeconfig for you.

```console
doctl kubernetes cluster create posthog-cluster --count=2 --size="s-2vcpu-4gb" --region=sfo3 --1-clicks=posthog
```
Comment on lines -40 to -46
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the best way to spin up PostHog & I use this command all the time. If someone is using digital-ocean already this is great, I'd propose we don't remove it. It's also especially great as it sets up your kubectl to point to the cluster too. That said it's going to be less commonly used so we can put it into a details box and hide by default.


### Configuring kubectl access

To monitor, debug, and maintain your PostHog instance, you should connect to your cluster using `kubectl`.

To do so, you should follow [these instructions](https://docs.digitalocean.com/products/kubernetes/how-to/connect-to-cluster/).

The simplest way to do this is with the following command:

```shell
doctl kubernetes cluster kubeconfig save your_posthog_cluster_name
```

### Accessing PostHog

After deploying PostHog to your Kubernetes cluster you can get the IP to connect to one of two ways:

<DOIPAddressSnippet />

Congrats, you now have a working PostHog instance!

You can now [integrate your development applications](/docs/integrate) with your PostHog deployment for testing purposes.

> Note: You will only be able to test web apps on HTTP such as those running on `localhost`.

But there's one more important step before using PostHog in production: securing your instance. See the next section for details.
Once the setup is completed, remember to [configure your `kubectl` access](https://docs.digitalocean.com/products/kubernetes/how-to/connect-to-cluster/) and [secure your 1-click installation](#securing-your-1-click-install).

### Securing your 1-click install

It's [not yet possible to provide parameters to DigitalOcean](https://github.com/digitalocean/marketplace-kubernetes/issues/230), so we need a post-install step to enable TLS.

> In order to do this you will need [kubectl](https://kubernetes.io/docs/tasks/tools/) and [helm](https://helm.sh/) installed.
It's unfortunately [not yet possible to provide parameters to DigitalOcean](https://github.com/digitalocean/marketplace-kubernetes/issues/230), so we need a post-install step to enable TLS.

#### 1. Lookup external IP
> In order to do this you will need [kubectl](https://kubernetes.io/docs/tasks/tools/) and [Helm](https://helm.sh/) installed.

As above, you can get the IP to connect to one of two ways:
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
#### 1. Lookup the IP address of the installation

<DOIPAddressSnippet />
<GetIPAddressSnippet />

#### 2. Set up DNS

Create an `A` record from your desired hostname to the external IP.
Create an `A` record from your desired hostname to the external IP we got above.

#### 3. Update PostHog

Expand All @@ -101,21 +67,27 @@ certManager:
enabled: true
```

Run the upgrade (note that if you used the UI for install, you'll need to follow the getting started guide to setup kubeconfig, if you skipped it earlier use the "Remind me how to do this" link on the Kubernetes cluster tab)
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
and then run:

```
```console
helm repo add posthog https://posthog.github.io/charts-clickhouse/
helm repo update
helm upgrade -f values.yaml --timeout 20m --namespace posthog posthog posthog/posthog
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
```

### Accessing PostHog

<GetUrlSnippet />

<PostInstallSnippet />

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The note about http only applies for when we haven't enabled TLS, so we should move that up.

## Manual install

Alternatively, to install the chart manually using [Helm >= v3](https://helm.sh/) follow these steps:
First, we need to set up a Kubernetes cluster (see the official DigitalOcean [documentation](https://docs.digitalocean.com/products/kubernetes/quickstart/) for more info).

### 1. Set up K8s cluster
First, we need to set up a Kubernetes Cluster. See [Kubernetes quickstart](https://docs.digitalocean.com/products/kubernetes/quickstart/). Note that the minimum total resource requirements to run PostHog are 4vcpu and 4G of memory.
<ClusterRequirementsSnippet />

#### 1. Chart configuration

Here's the minimal required `values.yaml` that we'll be using later. You can find an overview of the parameters that can be configured during installation under [configuration](/docs/self-host/deploy/configuration).
```yaml
Expand All @@ -131,23 +103,29 @@ kafka:
size: 20Gi
```

### 2. Install the chart
#### 2. Install the chart

<InstallingSnippet />

### 3. Lookup external IP
#### 3. Lookup the IP address of our installation

<GetIPAddressSnippet />

<DOIPAddressSnippet />
#### 4. Set up DNS

### 4. Set up DNS
Create an `A` record from your desired hostname to the external IP we got above.

Create an `A` record from your desired hostname to the external IP.
### Accessing PostHog

<GetUrlSnippet />

<PostInstallSnippet />
Comment on lines +119 to +121
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

http part drop completely here as we didn't really provide an option for an unsecure instance. But add it to the "I cannot connect to my PostHog instance after creation" section as it's relevant there.


## Troubleshooting

### I cannot connect to my PostHog instance after creation
<TryUnsecureSnippet />


## Upgrading the chart
<UpgradingSnippet />

Expand Down
9 changes: 7 additions & 2 deletions contents/docs/self-host/deploy/gcp.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,14 +4,19 @@ sidebarTitle: Google Cloud Platform
sidebar: Docs
showTitle: true
tags:
- gcp
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
- gcp
---

import ClusterRequirementsSnippet from './snippets/cluster-requirements'
import InstallingSnippet from './snippets/installing'
import UpgradingSnippet from './snippets/upgrading'
import UninstallingSnippet from './snippets/uninstalling'

First, we need to set up a Kubernetes Cluster. See [Google Cloud Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine/).
First, we need to set up a Kubernetes cluster (see the official GCP [documentation](https://cloud.google.com/kubernetes-engine/) for more info).
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as AWS & Azure - do we get the expanding volumes by default and if not how can we?

Copy link
Contributor

@tiina303 tiina303 Oct 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Plus it might be nice to mention in the DigitalOcean doc that we get that by default, so folks don't need to go investigate it. Note that we can document how to get expandable volumes for all of them in follow-up PRs.


<ClusterRequirementsSnippet />

## Chart configuration

Here's the minimal required `values.yaml` that we'll be using later. You can find an overview of the parameters that can be configured during installation under [configuration](/docs/self-host/deploy/configuration).
```yaml
Expand Down
8 changes: 3 additions & 5 deletions contents/docs/self-host/deploy/other.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,15 @@ sidebar: Docs
showTitle: true
---

import ClusterRequirementsSnippet from './snippets/cluster-requirements'
import InstallingSnippet from './snippets/installing'
import UpgradingSnippet from './snippets/upgrading'
import UninstallingSnippet from './snippets/uninstalling'
import TryUnsecureSnippet from './snippets/tryunsecure'

For all other platforms, we suggest setting up Kubernetes first and using the helm chart directly to deploy a PostHog instance with Nginx ingress controller.

## Prerequisites
- [Kubernetes](http://kubernetes.io) 1.19+
- [Helm](https://helm.sh) >= v3
For all the other platforms, setup your Kubernetes cluster (see the documentation of your platform for more info).

<ClusterRequirementsSnippet />

## Installing the chart

Expand Down
12 changes: 12 additions & 0 deletions contents/docs/self-host/deploy/snippets/cluster-requirements.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#### Cluster requirements
* Kubernetes version >= 1.19
* Ensure your cluster has enough resources to run PostHog (we suggest a total minimum of 4 vcpu & 8GB of memory).
* Ensure the `AllowVolumeExpansion` parameter is set to `True` in the storage class definition (this setting enables PVC resize)
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
<details>

tiina303 marked this conversation as resolved.
Show resolved Hide resolved
```console
$ kubectl get storageclass -o json | jq '.items[].allowVolumeExpansion'
true
```

</details>
10 changes: 0 additions & 10 deletions contents/docs/self-host/deploy/snippets/do-ip-address.mdx

This file was deleted.

6 changes: 6 additions & 0 deletions contents/docs/self-host/deploy/snippets/get-ip-address.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
```shell
export INGRESS_IP=$(kubectl get --namespace posthog ingress posthog -o jsonpath="{.status.loadBalancer.ingress[0].ip}") && \
echo "\n-----\n" && \
echo "Your PostHog IP address is: $INGRESS_IP" && \
echo "\n-----\n"
```
6 changes: 6 additions & 0 deletions contents/docs/self-host/deploy/snippets/get-url.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
```shell
export INGRESS_IP=$(kubectl get --namespace posthog ingress posthog -o jsonpath="{.status.loadBalancer.ingress[0].ip}") && \
echo "\n-----\n" && \
echo "Your PostHog installation is available at: http://$INGRESS_IP" && \
Comment on lines +2 to +4
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to get the URL here & I don't think this does that, but if nothing else the variable name is confusing

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which URL? This command will output something like:

-----

Your PostHog installation is available at: http://213.195.116.118

-----

Except for the output format I didn't change variable names or anything from the previous version. What do you mean "we need to get the URL here"?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If they already setup TLS they should access posthog via the hostname, not IP, e.g. http://app.posthog.com instead of http://104.22.58.181. Especially because it's possible to forbid direct IP access

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So there's two different things: and separately: access your PostHog instance if TLS was set up, where arguably we can just say navigate to your hostname

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 but for what I saw we never rendered the hostname, even before this PR

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would be a nice to have - definitely not a blocker for getting this PR in

echo "\n-----\n"
```
4 changes: 1 addition & 3 deletions contents/docs/self-host/deploy/snippets/installing.mdx
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
To install the chart with the release name `posthog` in `posthog` namespace, run the following:
To install the chart using [Helm >= v3](https://helm.sh/) with the release name `posthog` in `posthog` namespace, run the following:

```console
helm repo add posthog https://posthog.github.io/charts-clickhouse/
helm repo update
helm install -f values.yaml --timeout 20m --create-namespace --namespace posthog posthog posthog/posthog
```


2 changes: 2 additions & 0 deletions contents/docs/self-host/deploy/snippets/post-install.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Congrats, you now have a working PostHog instance! You can now [integrate your development applications](/docs/integrate) with your PostHog deployment for testing purposes.
tiina303 marked this conversation as resolved.
Show resolved Hide resolved
> Note: You will only be able to test web apps on HTTP such as those running on `localhost`.
11 changes: 5 additions & 6 deletions contents/docs/self-host/deploy/snippets/tryunsecure.mdx
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
As a troubleshooting tool, you can allow HTTP access by setting these values in your `values.yaml`, but we recommend always accessing PostHog via https.
import GetURLSnippet from './get-url'

As a troubleshooting tool, you can allow HTTP access by setting these values in your `values.yaml`, but we recommend always accessing PostHog via HTTPs.
```yaml
ingress:
nginx:
Expand All @@ -9,9 +11,6 @@ web:
secureCookies: false
```

After upgrading you can run the following to get the IP to access PostHog:
```console
export INGRESS_IP=$(kubectl get --namespace posthog ingress posthog -o jsonpath="{.status.loadBalancer.ingress[0].ip}")
echo "Visit http://$INGRESS_IP to use PostHog\!"
```
After upgrading you can run the following command to get the URL to access PostHog:

<GetURLSnippet />
8 changes: 4 additions & 4 deletions contents/docs/self-host/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ Getting a shiny, running production environment of PostHog is probably one the f

Lucky for you, our platform is incredibly easy to use and affordable to host with any provider. Below, we have several step-by-step guides outlining how to set up hosting on a variety of different services.

### **Deployment options**
### Deployment options

- [DigitalOcean](/docs/self-host/deploy/digital-ocean)
- [Google Cloud Platform](/docs/self-host/deploy/gcp)
- [AWS](/docs/self-host/deploy/aws)
- [Azure](/docs/self-host/deploy/azure)
- [Using Helm Charts](/docs/self-host/deploy/other)
- [DigitalOcean](/docs/self-host/deploy/digital-ocean)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sort alphabetically

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We actually sorted them that way on purpose. DigitalOcean is easiest to get started with and most transparent about pricing, but maybe we should explicitly say this here instead.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be subjective and not objective. Is it possible that our users find the pricing and setup of Azure or Oracle Cloud easier than DigitalOcean (maybe because they are already familiar with the platform).

I’m inclined to keep our docs as impartial as possible if you agree.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

- [Google Cloud Platform](/docs/self-host/deploy/gcp)
- [Other platforms](/docs/self-host/deploy/other)

## Configure

Expand Down
Binary file removed contents/images/do-ip-location.png
Binary file not shown.
Loading