-
Notifications
You must be signed in to change notification settings - Fork 16
Add initial version of ParadeDB extension #122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
46d2dc3
add initial version of ParadeDB extension
whummer 824122a
fix ParadeDB tests to use v2 API syntax
whummer fcbe268
[paradedb] Prefer more distinct connection credentials
purcell c86df6b
[paradedb] Also allow port to be overridden
purcell 5ed49e5
[paradedb] Additional project keywords
purcell b255207
[paradedb] Remove hallucinated project description
purcell 535795d
[paradedb] Fix hallucinated example
purcell a7aa974
[paradedb] Just use the quickstart example as a test
purcell 6c8686f
[paradedb] Apply formatter
purcell c0a9665
[paradedb] Add to main list of available extensions
purcell File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| name: LocalStack ParadeDB Extension Tests | ||
|
|
||
| on: | ||
| push: | ||
| paths: | ||
| - paradedb/** | ||
| branches: | ||
| - main | ||
| pull_request: | ||
| paths: | ||
| - .github/workflows/paradedb.yml | ||
| - paradedb/** | ||
| workflow_dispatch: | ||
|
|
||
| env: | ||
| LOCALSTACK_DISABLE_EVENTS: "1" | ||
| LOCALSTACK_AUTH_TOKEN: ${{ secrets.LOCALSTACK_AUTH_TOKEN }} | ||
|
|
||
| jobs: | ||
| integration-tests: | ||
| name: Run Integration Tests | ||
| runs-on: ubuntu-latest | ||
| timeout-minutes: 10 | ||
| steps: | ||
| - name: Checkout | ||
| uses: actions/checkout@v4 | ||
|
|
||
| - name: Setup LocalStack and extension | ||
| run: | | ||
| cd paradedb | ||
|
|
||
| docker pull localstack/localstack-pro & | ||
| docker pull paradedb/paradedb & | ||
| pip install localstack | ||
|
|
||
| make install | ||
| make lint | ||
| make dist | ||
| localstack extensions -v install file://$(ls ./dist/localstack_extension_paradedb-*.tar.gz) | ||
|
|
||
| DEBUG=1 localstack start -d | ||
| localstack wait | ||
|
|
||
| - name: Run integration tests | ||
| run: | | ||
| cd paradedb | ||
| make test | ||
|
|
||
| - name: Print logs | ||
| if: always() | ||
| run: | | ||
| localstack logs | ||
| localstack stop |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| { | ||
| description = "localstack-extensions"; | ||
|
|
||
| inputs = { | ||
| nixpkgs.url = "nixpkgs/nixpkgs-unstable"; | ||
| }; | ||
|
|
||
| outputs = { self, nixpkgs }@inputs: | ||
| ( | ||
| let | ||
| forAllSystems = nixpkgs.lib.genAttrs nixpkgs.lib.platforms.all; | ||
| in | ||
| { | ||
| devShell = forAllSystems (system: | ||
| let pkgs = import nixpkgs { inherit system; }; in | ||
| pkgs.mkShell { | ||
| buildInputs = with pkgs; [ uv python311 python311Packages.pip ty ]; | ||
| } | ||
| ); | ||
| } | ||
| ); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| .venv | ||
| dist | ||
| build | ||
| **/*.egg-info | ||
| .eggs |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,48 @@ | ||
| VENV_BIN = python3 -m venv | ||
| VENV_DIR ?= .venv | ||
| VENV_ACTIVATE = $(VENV_DIR)/bin/activate | ||
| VENV_RUN = . $(VENV_ACTIVATE) | ||
| TEST_PATH ?= tests | ||
|
|
||
| usage: ## Shows usage for this Makefile | ||
| @cat Makefile | grep -E '^[a-zA-Z_-]+:.*?## .*$$' | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-15s\033[0m %s\n", $$1, $$2}' | ||
|
|
||
| venv: $(VENV_ACTIVATE) | ||
|
|
||
| $(VENV_ACTIVATE): pyproject.toml | ||
| test -d .venv || $(VENV_BIN) .venv | ||
| $(VENV_RUN); pip install --upgrade pip setuptools plux | ||
| $(VENV_RUN); pip install -e .[dev] | ||
| touch $(VENV_DIR)/bin/activate | ||
|
|
||
| clean: | ||
| rm -rf .venv/ | ||
| rm -rf build/ | ||
| rm -rf .eggs/ | ||
| rm -rf *.egg-info/ | ||
|
|
||
| install: venv ## Install dependencies | ||
| $(VENV_RUN); python -m plux entrypoints | ||
|
|
||
| dist: venv ## Create distribution | ||
| $(VENV_RUN); python -m build | ||
|
|
||
| publish: clean-dist venv dist ## Publish extension to pypi | ||
| $(VENV_RUN); pip install --upgrade twine; twine upload dist/* | ||
|
|
||
| entrypoints: venv ## Generate plugin entrypoints for Python package | ||
| $(VENV_RUN); python -m plux entrypoints | ||
|
|
||
| format: ## Run ruff to format the codebase | ||
| $(VENV_RUN); python -m ruff format .; make lint | ||
|
|
||
| lint: ## Run ruff to lint the codebase | ||
| $(VENV_RUN); python -m ruff check --output-format=full . | ||
|
|
||
| test: ## Run integration tests (requires LocalStack running with the Extension installed) | ||
| $(VENV_RUN); pytest $(PYTEST_ARGS) $(TEST_PATH) | ||
|
|
||
| clean-dist: clean | ||
| rm -rf dist/ | ||
|
|
||
| .PHONY: clean clean-dist dist install publish usage venv format test |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,95 @@ | ||
| ParadeDB on LocalStack | ||
| ====================== | ||
|
|
||
| This repo contains a [LocalStack Extension](https://github.com/localstack/localstack-extensions) that facilitates developing [ParadeDB](https://www.paradedb.com)-based applications locally. | ||
|
|
||
| ParadeDB is an Elasticsearch alternative built on Postgres. It provides full-text search with BM25 scoring, hybrid search combining semantic and keyword search, and real-time analytics capabilities. | ||
|
|
||
| After installing the extension, a ParadeDB server instance will become available and can be accessed using standard PostgreSQL clients. | ||
|
|
||
| ## Connection Details | ||
|
|
||
| Once the extension is running, you can connect to ParadeDB using any PostgreSQL client with the following default credentials: | ||
|
|
||
| - **Host**: `localhost` (or the Docker host if running in a container) | ||
| - **Port**: `5432` (mapped from the container) | ||
| - **Database**: `mydatabase` | ||
| - **Username**: `myuser` | ||
| - **Password**: `mypassword` | ||
|
|
||
| Example connection using `psql`: | ||
| ```bash | ||
| psql -h localhost -p 5432 -U myuser -d mydatabase | ||
| ``` | ||
|
|
||
| Example connection using Python: | ||
| ```python | ||
| import psycopg2 | ||
|
|
||
| conn = psycopg2.connect( | ||
| host="localhost", | ||
| port=5432, | ||
| database="mydatabase", | ||
| user="myuser", | ||
| password="mypassword" | ||
| ) | ||
| ``` | ||
|
|
||
| ## ParadeDB Features | ||
|
|
||
| ParadeDB includes the **pg_search** extension, for both search and | ||
| analytics workloads. | ||
|
|
||
| Example of BM25 scoring, from the excellent [quickstart guide](https://docs.paradedb.com/documentation/getting-started/quickstart): | ||
|
|
||
| ```sql | ||
| CALL paradedb.create_bm25_test_table( | ||
| schema_name => 'public', | ||
| table_name => 'mock_items' | ||
| ); | ||
|
|
||
| CREATE INDEX search_idx ON mock_items | ||
| USING bm25 (id, description, category, rating, in_stock, created_at, metadata, weight_range) | ||
| WITH (key_field='id'); | ||
|
|
||
| SELECT description, pdb.score(id) | ||
| FROM mock_items | ||
| WHERE description ||| 'running shoes' AND rating > 2 | ||
| ORDER BY score DESC | ||
| LIMIT 5; | ||
| ``` | ||
|
|
||
| ## Configuration | ||
|
|
||
| The following environment variables can be passed to the LocalStack container to configure the extension: | ||
|
|
||
| * `PARADEDB_POSTGRES_USER`: PostgreSQL username (default: `myuser`) | ||
| * `PARADEDB_POSTGRES_PASSWORD`: PostgreSQL password (default: `mypassword`) | ||
| * `PARADEDB_POSTGRES_DB`: Default database name (default: `mydatabase`) | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| * Docker | ||
| * LocalStack Pro (free trial available) | ||
| * `localstack` CLI | ||
| * `make` | ||
|
|
||
| ## Install from GitHub repository | ||
|
|
||
| This extension can be installed directly from this Github repo via: | ||
|
|
||
| ```bash | ||
| localstack extensions install "git+https://github.com/localstack/localstack-extensions.git#egg=localstack-extension-paradedb&subdirectory=paradedb" | ||
| ``` | ||
|
|
||
| ## Install local development version | ||
|
|
||
| Please refer to the docs [here](https://github.com/localstack/localstack-extensions?tab=readme-ov-file#start-localstack-with-the-extension) for instructions on how to start the extension in developer mode. | ||
|
|
||
| ## Change Log | ||
|
|
||
| * `0.1.0`: Initial version of the extension | ||
|
|
||
| ## License | ||
|
|
||
| The code in this repo is available under the Apache 2.0 license. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| name = "localstack_paradedb" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,70 @@ | ||
| import os | ||
| import logging | ||
|
|
||
| from localstack_paradedb.utils.docker import DatabaseDockerContainerExtension | ||
|
|
||
| LOG = logging.getLogger(__name__) | ||
|
|
||
| # Environment variables for configuration | ||
| ENV_POSTGRES_USER = "PARADEDB_POSTGRES_USER" | ||
| ENV_POSTGRES_PASSWORD = "PARADEDB_POSTGRES_PASSWORD" | ||
| ENV_POSTGRES_DB = "PARADEDB_POSTGRES_DB" | ||
| ENV_POSTGRES_PORT = "PARADEDB_POSTGRES_PORT" | ||
|
|
||
| # Default values | ||
| DEFAULT_POSTGRES_USER = "myuser" | ||
| DEFAULT_POSTGRES_PASSWORD = "mypassword" | ||
| DEFAULT_POSTGRES_DB = "mydatabase" | ||
| DEFAULT_POSTGRES_PORT = 5432 | ||
|
|
||
|
|
||
| class ParadeDbExtension(DatabaseDockerContainerExtension): | ||
| name = "paradedb" | ||
|
|
||
| # Name of the Docker image to spin up | ||
| DOCKER_IMAGE = "paradedb/paradedb" | ||
|
|
||
| def __init__(self): | ||
| # Get configuration from environment variables | ||
| postgres_user = os.environ.get(ENV_POSTGRES_USER, DEFAULT_POSTGRES_USER) | ||
| postgres_password = os.environ.get( | ||
| ENV_POSTGRES_PASSWORD, DEFAULT_POSTGRES_PASSWORD | ||
| ) | ||
| postgres_db = os.environ.get(ENV_POSTGRES_DB, DEFAULT_POSTGRES_DB) | ||
| postgres_port = int(os.environ.get(ENV_POSTGRES_PORT, DEFAULT_POSTGRES_PORT)) | ||
|
|
||
| # Environment variables to pass to the container | ||
| env_vars = { | ||
| "POSTGRES_USER": postgres_user, | ||
| "POSTGRES_PASSWORD": postgres_password, | ||
| "POSTGRES_DB": postgres_db, | ||
| } | ||
|
|
||
| super().__init__( | ||
| image_name=self.DOCKER_IMAGE, | ||
| container_ports=[postgres_port], | ||
| env_vars=env_vars, | ||
| ) | ||
|
|
||
| # Store configuration for connection info | ||
| self.postgres_user = postgres_user | ||
| self.postgres_password = postgres_password | ||
| self.postgres_db = postgres_db | ||
| self.postgres_port = postgres_port | ||
|
|
||
| def get_connection_info(self) -> dict: | ||
| """Return connection information for ParadeDB.""" | ||
| info = super().get_connection_info() | ||
| info.update( | ||
| { | ||
| "database": self.postgres_db, | ||
| "user": self.postgres_user, | ||
| "password": self.postgres_password, | ||
| "port": self.postgres_port, | ||
| "connection_string": ( | ||
| f"postgresql://{self.postgres_user}:{self.postgres_password}" | ||
| f"@{self.container_host}:{self.postgres_port}/{self.postgres_db}" | ||
| ), | ||
| } | ||
| ) | ||
| return info |
Empty file.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.