Skip to content

Commit

Permalink
Add how to copy files from Kubernetes pods. (airbytehq#7238)
Browse files Browse the repository at this point in the history
* Add how to copy files from Kubernetes pods.

* Add instructions on how to browse.

* Format.

* This is needed.

Co-authored-by: Christophe Duong <christophe.duong@gmail.com>
  • Loading branch information
2 people authored and schlattk committed Jan 4, 2022
1 parent 9d98945 commit f9ca8e0
Showing 1 changed file with 22 additions and 0 deletions.
22 changes: 22 additions & 0 deletions docs/operator-guides/browsing-output-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,28 @@ docker cp airbyte-server:/tmp/workspace/9/2/catalog.json .
cat catalog.json
```

### Browsing on Kubernetes

If you are running on Kubernetes, use the following commands instead to browsing and copy the files to your local.

To browse, identify the pod you are interested in and exec into it. You will be presented with a terminal that will accept normal linux commands e.g ls.
```bash
kubectl exec -it <pod name> -n <namespace pod is in> -c main bash
e.g.
kubectl exec -it destination-bigquery-worker-3607-0-chlle -n jobs -c main bash
root@destination-bigquery-worker-3607-0-chlle:/config# ls
FINISHED_UPLOADING destination_catalog.json destination_config.json
```

To copy the file on to your local in order to preserve it's contents:
```bash
kubectl cp <namespace pods are in>/<normalisation-pod-name>:/config/destination_catalog.json ./catalog.json
e.g.
kubectl cp jobs/normalization-worker-3605-0-sxtox:/config/destination_catalog.json ./catalog.json
cat ./catalog.json
```


## CSV or JSON local Destinations: Check local data folder

If you setup a pipeline using one of the local File based destinations \(CSV or JSON\), Airbyte is writing the resulting files containing the data in the special `/local/` directory in the container. By default, this volume is mounted from `/tmp/airbyte_local` on the host machine. So you need to navigate to this [local folder](file:///tmp/airbyte_local/) on the filesystem of the machine running the Airbyte deployment to retrieve the local data files.
Expand Down

0 comments on commit f9ca8e0

Please sign in to comment.