Skip to content

Commit

Permalink
📝 Add warning for working with local files and Docker Desktop (/tmp
Browse files Browse the repository at this point in the history
… and `/private`) (#21004)
  • Loading branch information
sspaeti committed Jan 5, 2023
1 parent 50916e7 commit 26053ef
Show file tree
Hide file tree
Showing 6 changed files with 33 additions and 1 deletion.
6 changes: 6 additions & 0 deletions docs/integrations/destinations/local-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ This destination is meant to be used on a local workstation and won't work on Ku

This destination writes data to a directory on the _local_ filesystem on the host running Airbyte. By default, data is written to `/tmp/airbyte_local`. To change this location, modify the `LOCAL_ROOT` environment variable for Airbyte.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

### Sync Overview

#### Output schema
Expand Down
7 changes: 7 additions & 0 deletions docs/integrations/destinations/local-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,13 @@ By default, the `LOCAL_ROOT` env variable in the `.env` file is set `/tmp/airbyt

The local mount is mounted by Docker onto `LOCAL_ROOT`. This means the `/local` is substituted by `/tmp/airbyte_local` by default.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::


### Example:

* If `destination_path` is set to `/local/cars/models`
Expand Down
8 changes: 7 additions & 1 deletion docs/integrations/destinations/sqlite.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,12 @@ This destination is meant to be used on a local workstation and won't work on Ku

This destination writes data to a file on the _local_ filesystem on the host running Airbyte. By default, data is written to `/tmp/airbyte_local`. To change this location, modify the `LOCAL_ROOT` environment variable for Airbyte.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

### Sync Overview

#### Output schema
Expand Down Expand Up @@ -68,4 +74,4 @@ Note: If you are running Airbyte on Windows with Docker backed by WSL2, you have

| Version | Date | Pull Request | Subject |
| :--- | :--- | :--- | :--- |
| 0.1.0 | 2022-07-25 | [15018](https://github.com/airbytehq/airbyte/pull/15018) | New SQLite destination |
| 0.1.0 | 2022-07-25 | [15018](https://github.com/airbytehq/airbyte/pull/15018) | New SQLite destination |
1 change: 1 addition & 0 deletions docs/integrations/sources/file.md
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,7 @@ Please see (or add) more at `airbyte-integrations/connectors/source-file/integra
In order to read large files from a remote location, this connector uses the [smart_open](https://pypi.org/project/smart-open/) library. However, it is possible to switch to either [GCSFS](https://gcsfs.readthedocs.io/en/latest/) or [S3FS](https://s3fs.readthedocs.io/en/latest/) implementations as it is natively supported by the `pandas` library. This choice is made possible through the optional `reader_impl` parameter.

- Note that for local filesystem, the file probably have to be stored somewhere in the `/tmp/airbyte_local` folder with the same limitations as the [CSV Destination](../destinations/local-csv.md) so the `URL` should also starts with `/local/`.
- Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.
- The JSON implementation needs to be tweaked in order to produce more complex catalog and is still in an experimental state: Simple JSON schemas should work at this point but may not be well handled when there are multiple layers of nesting.

## Changelog
Expand Down
6 changes: 6 additions & 0 deletions docs/operator-guides/browsing-output-logs.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,12 @@ cat ./catalog.json

If you setup a pipeline using one of the local File based destinations \(CSV or JSON\), Airbyte is writing the resulting files containing the data in the special `/local/` directory in the container. By default, this volume is mounted from `/tmp/airbyte_local` on the host machine. So you need to navigate to this [local folder](file:///tmp/airbyte_local/) on the filesystem of the machine running the Airbyte deployment to retrieve the local data files.

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

Or, you can also run through docker commands as proxy:

```bash
Expand Down
6 changes: 6 additions & 0 deletions docs/quickstart/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,12 @@ The destination we are creating is a simple JSON line file, meaning that it will

The resulting files will be located in `/tmp/airbyte_local/json_data`

:::caution

Please make sure that Docker Desktop has access to `/tmp` (and `/private` on a MacOS, as /tmp has a symlink that points to /private. It will not work otherwise). You allow it with "File sharing" in `Settings -> Resources -> File sharing -> add the one or two above folder` and hit the "Apply & restart" button.

:::

To set it up, just follow the instructions on the screenshot below.

:::info
Expand Down

0 comments on commit 26053ef

Please sign in to comment.