Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH tunnel support for S3Store #882

Merged
merged 19 commits into from Nov 21, 2023
Merged
Show file tree
Hide file tree
Changes from 16 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
56 changes: 56 additions & 0 deletions docs/getting_started/using_ssh_tunnel.md
@@ -0,0 +1,56 @@
# Using `SSHTunnel` to conect to remote database

One of the typical scenarios to use `maggma` is to connect to a remote database that is behind a firewall and thus cannot be accessed directly from your local computer (as shown below, [image credits](https://github.com/pahaz/sshtunnel/)).

In this case, you can use `SSHTunnel` to first connect to the remote server, and then connect to the database from the server.

```
----------------------------------------------------------------------

|
-------------+ | +----------+ +---------
LOCAL | | | REMOTE | | PRIVATE
COMPUTER | <== SSH ========> | SERVER | <== local ==> | SERVER
-------------+ | +----------+ +---------
|
FIREWALL (only port 22 is open)

----------------------------------------------------------------------

Note, the `local` indicates that the connction to the PRIVATE SERVER can only be made from the REMOTE SERVER.
```


Below is an example of how to use `SSHTunnel` to connect to an AWS `S3Store` hosted on a private server.

Let's assume that, from you local computer, you can ssh to the remote server using the following command with your credentials (e.g. <USER_CREDENTIAL>):

```bash
ssh <USERNAME>@<REMOTE_SERVER_ADDRESS>
```

and then from the remote server, you can access your database using, e.g., the following information:
```
private_server_address: COMPUTE_NODE_1
private_server_port: 9000
```

You can create an `SSHTunnel` object as follows:

```python
from maggma.stores.ssh_tunnel import SSHTunnel

tunnel = SSHTunnel(
tunnel_server_address = "<REMOTE_SERVER_ADDRESS>:22",
username = "<USERNAME>",
password= "<USER_CREDENTIAL>",
remote_server_address = "COMPUTE_NODE_1:9000",
local_port = 9000,
)
```
and then pass it to the `S3Store` to connect to the database.
mjwen marked this conversation as resolved.
Show resolved Hide resolved
By doing so, you can access the database at the localhost address `http://127.0.0.1:9000` from your local computer as if it is hosted on your local computer.

The arguments of the `SSHTunnel` are self-explanatory, but `local_port` needs more explanation. We assume that on the local computer, we want t o connect to the localhost address `http://127.0.0.1`, so we do not need to provide the address, but only the port number (`9000` in this case.)
rkingsbury marked this conversation as resolved.
Show resolved Hide resolved

In essence, `SSHTunnel` allows you to connect to `COMPUTE_NODE_1:9000` on the private server from `http://127.0.0.1:9000` on your local computer.
2 changes: 2 additions & 0 deletions docs/reference/stores.md
Expand Up @@ -9,3 +9,5 @@
::: maggma.stores.advanced_stores

::: maggma.stores.compound_stores

::: maggma.stores.ssh_tunnel
1 change: 1 addition & 0 deletions mkdocs.yml
Expand Up @@ -17,6 +17,7 @@ nav:
- Working with MapBuilder: getting_started/map_builder.md
- Working with GroupBuilder: getting_started/group_builder.md
- Setting up MongoDB: getting_started/mongodb.md
- Using SSHTunnel: getting_started/using_ssh_tunnel.md
- Reference:
Core:
Store: reference/core_store.md
Expand Down