Skip to content

Commit

Permalink
Add how to copy files from Kubernetes pods. (#7238)
Browse files Browse the repository at this point in the history
* Add how to copy files from Kubernetes pods.

* Add instructions on how to browse.

* Format.

* This is needed.

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
  • Loading branch information
davinchia and ChristopheDuong committed Nov 18, 2021
1 parent 8e0e0b4 commit 7b54480
Showing 1 changed file with 22 additions and 0 deletions.
22 changes: 22 additions & 0 deletions docs/operator-guides/browsing-output-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,28 @@ docker cp airbyte-server:/tmp/workspace/9/2/catalog.json .
cat catalog.json
```

### Browsing on Kubernetes

If you are running on Kubernetes, use the following commands instead to browsing and copy the files to your local.

To browse, identify the pod you are interested in and exec into it. You will be presented with a terminal that will accept normal linux commands e.g ls.
```bash
kubectl exec -it <pod name> -n <namespace pod is in> -c main bash
e.g.
kubectl exec -it destination-bigquery-worker-3607-0-chlle -n jobs -c main bash
root@destination-bigquery-worker-3607-0-chlle:/config# ls
FINISHED_UPLOADING destination_catalog.json destination_config.json
```

To copy the file on to your local in order to preserve it's contents:
```bash
kubectl cp <namespace pods are in>/<normalisation-pod-name>:/config/destination_catalog.json ./catalog.json
e.g.
kubectl cp jobs/normalization-worker-3605-0-sxtox:/config/destination_catalog.json ./catalog.json
cat ./catalog.json
```


## CSV or JSON local Destinations: Check local data folder

If you setup a pipeline using one of the local File based destinations \(CSV or JSON\), Airbyte is writing the resulting files containing the data in the special `/local/` directory in the container. By default, this volume is mounted from `/tmp/airbyte_local` on the host machine. So you need to navigate to this [local folder](file:///tmp/airbyte_local/) on the filesystem of the machine running the Airbyte deployment to retrieve the local data files.
Expand Down

0 comments on commit 7b54480

Please sign in to comment.