diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d6ca0304..c7159cc9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -25,27 +25,150 @@ pre-commit run --all-files ## Running tests -To create a test environment, do the following: -``` -pip install -e ".[dev]" +### Prerequisites + +Before running tests, ensure you have: +- **Docker installed and running** (for automatic test database management) +- Test dependencies installed: `pip install -e ".[test]"` or `pip install -e ".[dev]"` for all development dependencies + +The `docker` Python package is required for the test framework to manage Docker containers automatically. + +### Installation + +To create a test environment: +```bash +pip install -e ".[dev]" # All development dependencies (recommended) ``` Or if you only need specific dependency groups: -``` -pip install -e ".[test]" # Just testing dependencies -pip install -e ".[docs]" # Just documentation dependencies +```bash +pip install -e ".[test]" # Just testing dependencies +pip install -e ".[docs]" # Just documentation dependencies +pip install -e ".[build]" # Just build dependencies +pip install -e ".[examples]" # Just example/demo dependencies ``` -If you have Docker installed, you can run the tests as follows. Note that -you should run the tests using both standard protocol and Data API (HTTP): -``` +### Basic Testing + +The test framework provides **automatic Docker container management**. When you run tests without setting `SINGLESTOREDB_URL`, the framework will: + +1. Automatically start a SingleStore Docker container (`ghcr.io/singlestore-labs/singlestoredb-dev`) +2. Allocate dynamic ports to avoid conflicts (MySQL, Data API, Studio) +3. Wait for the container to be ready +4. Run all tests against the container +5. Clean up the container after tests complete + +#### Standard MySQL Protocol Tests +```bash +# Run all tests (auto-starts Docker container) pytest -v singlestoredb/tests -USE_DATA_API=1 -v singlestoredb/tests + +# Run with coverage +pytest -v --cov=singlestoredb --pyargs singlestoredb.tests + +# Run single test file +pytest singlestoredb/tests/test_basics.py + +# Run without management API tests +pytest -v -m 'not management' singlestoredb/tests ``` -If you need to run against a specific server version, you can specify -the URL of that server: +#### Data API Tests + +The SDK supports testing via SingleStore's **Data API** (port 9000) instead of the MySQL protocol (port 3306). This mode uses a **dual-URL system**: + +- `SINGLESTOREDB_URL`: Set to HTTP Data API endpoint (port 9000) for test operations +- `SINGLESTOREDB_INIT_DB_URL`: Automatically set to MySQL endpoint (port 3306) for setup operations + +**Why dual URLs?** Some setup operations like `SET GLOBAL` and `USE database` commands don't work over the HTTP Data API, so they're routed through the MySQL protocol automatically. + +Enable HTTP Data API testing: +```bash +# Run tests via HTTP Data API +USE_DATA_API=1 pytest -v singlestoredb/tests +``` + +**Known Limitations in HTTP Data API Mode:** +- `USE database` command is not supported (some tests will be skipped) +- Setup operations requiring `SET GLOBAL` are automatically routed to MySQL protocol + +#### Testing Against an Existing Server + +If you have a running SingleStore instance, you can test against it by setting `SINGLESTOREDB_URL`. The Docker container will not be started. + +```bash +# Test against MySQL protocol +SINGLESTOREDB_URL=user:password@host:3306 pytest -v singlestoredb/tests + +# Test against Data API +SINGLESTOREDB_INIT_DB_URL=user:password@host:3306 \ + SINGLESTOREDB_URL=http://user:password@host:9000 \ + pytest -v singlestoredb/tests ``` -SINGLESTOREDB_URL=user:pw@127.0.0.1:3306 pytest -v singlestoredb/tests -SINGLESTOREDB_URL=http://user:pw@127.0.0.1:8090 pytest -v singlestoredb/tests + +### Docker Container Details + +When the test framework starts a Docker container automatically: + +- **Container name**: `singlestoredb-test-{worker}-{uuid}` (supports parallel test execution) +- **Port mappings**: + - MySQL protocol: Random available port → Container port 3306 + - Data API (HTTP): Random available port → Container port 9000 + - Studio: Random available port → Container port 8080 +- **License**: Uses `SINGLESTORE_LICENSE` environment variable if set, otherwise runs without license +- **Cleanup**: Container is automatically removed after tests complete + +### Environment Variables + +The following environment variables control test behavior: + +- **`SINGLESTOREDB_URL`**: Database connection URL. If not set, a Docker container is started automatically. + - MySQL format: `user:password@host:3306` + - HTTP format: `http://user:password@host:9000` + +- **`USE_DATA_API`**: Set to `1`, `true`, or `on` to run tests via HTTP Data API instead of MySQL protocol. + - Automatically sets up the dual-URL system + - Example: `USE_DATA_API=1 pytest -v singlestoredb/tests` + +- **`SINGLESTOREDB_INIT_DB_URL`**: MySQL connection URL for setup operations (auto-set in HTTP Data API mode). Used for operations that require MySQL protocol even when testing via HTTP. + +- **`SINGLESTORE_LICENSE`**: Optional license key for Docker container. If not provided, container runs without a license. + +- **`SINGLESTOREDB_PURE_PYTHON`**: Set to `1` to disable C acceleration and test in pure Python mode. + +- **`SINGLESTOREDB_MANAGEMENT_TOKEN`**: Management API token for testing management features (mark tests with `@pytest.mark.management`). + +### Testing Best Practices + +1. **Test both protocols**: Always run tests with both MySQL protocol and HTTP Data API before submitting: + ```bash + pytest -v singlestoredb/tests + USE_DATA_API=1 pytest -v singlestoredb/tests + ``` + +2. **Pure Python testing**: Test without C acceleration to ensure compatibility: + ```bash + SINGLESTOREDB_PURE_PYTHON=1 pytest -v singlestoredb/tests + ``` + +3. **Management API tests**: These require a management token and are marked with `@pytest.mark.management`. + +### Examples + +```bash +# Standard workflow - test both protocols +pytest -v singlestoredb/tests +USE_DATA_API=1 pytest -v singlestoredb/tests + +# Test single module with coverage +pytest -v --cov=singlestoredb.connection singlestoredb/tests/test_connection.py + +# Test UDF functionality +pytest singlestoredb/tests/test_udf.py + +# Test against specific server (skips Docker) +SINGLESTOREDB_URL=admin:pass@localhost:3306 pytest -v singlestoredb/tests + +# Debug mode with verbose output +pytest -vv -s singlestoredb/tests/test_basics.py ``` diff --git a/pyproject.toml b/pyproject.toml index 06234140..2a04280b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -47,6 +47,7 @@ vectorstore = ["singlestore-vectorstore>=0.1.2"] test = [ "coverage", "dash", + "docker", "fastapi", "ipython", "jupysql", diff --git a/singlestoredb/pytest.py b/singlestoredb/pytest.py index 22efd68d..1170efea 100644 --- a/singlestoredb/pytest.py +++ b/singlestoredb/pytest.py @@ -86,9 +86,27 @@ def node_name() -> Iterator[str]: class _TestContainerManager(): - """Manages the setup and teardown of a SingleStoreDB Dev Container""" + """Manages the setup and teardown of a SingleStoreDB Dev Container + + If SINGLESTOREDB_URL environment variable is set, the manager will use + the existing server instead of starting a Docker container. This allows + tests to run against either an existing server or an automatically + managed Docker container. + """ def __init__(self) -> None: + # Check if SINGLESTOREDB_URL is already set - if so, use existing server + self.existing_url = os.environ.get('SINGLESTOREDB_URL') + self.use_existing = self.existing_url is not None + + if self.use_existing: + logger.info('Using existing SingleStore server from SINGLESTOREDB_URL') + self.url = self.existing_url + # No need to initialize Docker-related attributes + return + + logger.info('SINGLESTOREDB_URL not set, will start Docker container') + # Generate unique container name using UUID and worker ID worker = os.environ.get('PYTEST_XDIST_WORKER', 'master') unique_id = uuid.uuid4().hex[:8] @@ -96,11 +114,15 @@ def __init__(self) -> None: self.dev_image_name = 'ghcr.io/singlestore-labs/singlestoredb-dev' - assert 'SINGLESTORE_LICENSE' in os.environ, 'SINGLESTORE_LICENSE not set' + # Use SINGLESTORE_LICENSE from environment, or empty string as fallback + # Empty string works for the client SDK + license = os.environ.get('SINGLESTORE_LICENSE', '') + if not license: + logger.info('SINGLESTORE_LICENSE not set, using empty string') self.root_password = 'Q8r4D7yXR8oqn' self.environment_vars = { - 'SINGLESTORE_LICENSE': None, + 'SINGLESTORE_LICENSE': license, 'ROOT_PASSWORD': f"\"{self.root_password}\"", 'SINGLESTORE_SET_GLOBAL_DEFAULT_PARTITIONS_PER_LEAF': '1', } @@ -111,12 +133,23 @@ def __init__(self) -> None: self.studio_port = _find_free_port() self.ports = [ (self.mysql_port, '3306'), # External port -> Internal port - (self.http_port, '8080'), - (self.studio_port, '9000'), + (self.studio_port, '8080'), # Studio + (self.http_port, '9000'), # Data API ] self.url = f'root:{self.root_password}@127.0.0.1:{self.mysql_port}' + @property + def http_connection_url(self) -> Optional[str]: + """HTTP connection URL for the SingleStoreDB server using Data API.""" + if self.use_existing: + # If using existing server, HTTP URL not available from manager + return None + return ( + f'singlestoredb+http://root:{self.root_password}@' + f'127.0.0.1:{self.http_port}' + ) + def _container_exists(self) -> bool: """Check if a container with this name already exists.""" try: @@ -169,11 +202,16 @@ def start(self) -> None: f'{self.http_port}, {self.studio_port}', ) try: - license = os.environ['SINGLESTORE_LICENSE'] + license = os.environ.get('SINGLESTORE_LICENSE', '') env = { 'SINGLESTORE_LICENSE': license, } - subprocess.check_call(command, shell=True, env=env) + # Capture output to avoid printing the container ID hash + subprocess.check_call( + command, shell=True, env=env, + stdout=subprocess.DEVNULL, + ) + except Exception as e: logger.exception(e) raise RuntimeError( @@ -250,30 +288,51 @@ def stop(self) -> None: logger.info('Cleaning up SingleStore DB dev container') logger.debug('Stopping container') try: - subprocess.check_call(f'docker stop {self.container_name}', shell=True) + subprocess.check_call( + f'docker stop {self.container_name}', + shell=True, + stdout=subprocess.DEVNULL, + ) + except Exception as e: logger.exception(e) raise RuntimeError('Failed to stop container.') from e logger.debug('Removing container') try: - subprocess.check_call(f'docker rm {self.container_name}', shell=True) + subprocess.check_call( + f'docker rm {self.container_name}', + shell=True, + stdout=subprocess.DEVNULL, + ) + except Exception as e: logger.exception(e) - raise RuntimeError('Failed to stop container.') from e + raise RuntimeError('Failed to remove container.') from e @pytest.fixture(scope='session') def singlestoredb_test_container( execution_mode: ExecutionMode, ) -> Iterator[_TestContainerManager]: - """Sets up and tears down the test container""" + """Sets up and tears down the test container + + If SINGLESTOREDB_URL is set in the environment, uses the existing server + and skips Docker container lifecycle management. Otherwise, automatically + starts a Docker container for testing. + """ if not isinstance(execution_mode, ExecutionMode): raise TypeError(f"Invalid execution mode '{execution_mode}'") container_manager = _TestContainerManager() + # If using existing server, skip all Docker lifecycle management + if container_manager.use_existing: + logger.info('Using existing server, skipping Docker container lifecycle') + yield container_manager + return + # In sequential operation do all the steps if execution_mode == ExecutionMode.SEQUENTIAL: logger.debug('Not distributed') diff --git a/singlestoredb/tests/conftest.py b/singlestoredb/tests/conftest.py new file mode 100644 index 00000000..681c2fe5 --- /dev/null +++ b/singlestoredb/tests/conftest.py @@ -0,0 +1,216 @@ +#!/usr/bin/env python +"""Pytest configuration for singlestoredb tests + +This module sets up automatic Docker container management for tests. +It works with both pytest-style and unittest-style tests. + +The conftest automatically: +1. Checks if SINGLESTOREDB_URL is set in the environment +2. If not set, starts a SingleStore Docker container +3. Sets SINGLESTOREDB_URL for all tests to use +4. Cleans up the container when tests complete + +Environment Variables: + - SINGLESTOREDB_URL: If set, tests will use this existing server instead + of starting a Docker container. This allows testing + against a specific server instance. + - USE_DATA_API: If set to 1/true/on, tests will use HTTP Data API + instead of MySQL protocol. When set, SINGLESTOREDB_URL + will be set to the HTTP URL, and SINGLESTOREDB_INIT_DB_URL + will be set to the MySQL URL for setup operations. + - SINGLESTORE_LICENSE: Optional. License key for Docker container. If not + set, an empty string is used as fallback. + +Available Fixtures: + - singlestoredb_test_container: Manages Docker container lifecycle + - singlestoredb_connection: Provides a connection to the test server + - singlestoredb_tempdb: Creates a temporary test database with cursor +""" +import logging +import os +from collections.abc import Iterator +from typing import Optional + +import pytest + +from singlestoredb.pytest import _TestContainerManager +from singlestoredb.pytest import execution_mode # noqa: F401 +from singlestoredb.pytest import ExecutionMode # noqa: F401 +from singlestoredb.pytest import name_allocator # noqa: F401 +from singlestoredb.pytest import node_name # noqa: F401 +from singlestoredb.pytest import singlestoredb_connection # noqa: F401 +from singlestoredb.pytest import singlestoredb_tempdb # noqa: F401 + + +logger = logging.getLogger(__name__) + +# Global container manager instance +_container_manager: Optional[_TestContainerManager] = None + + +def pytest_configure(config: pytest.Config) -> None: + """ + Pytest hook that runs before test collection. + + This ensures the Docker container is started (if needed) before any + test modules are imported. Some test modules try to get connection + parameters at import time, so we need the environment set up early. + """ + global _container_manager + + # Prevent double initialization - pytest_configure can be called multiple times + if _container_manager is not None: + logger.debug('pytest_configure already called, skipping') + return + + if 'SINGLESTOREDB_URL' not in os.environ: + print('\n' + '=' * 70) + print('Starting SingleStoreDB Docker container...') + print('This may take a moment...') + print('=' * 70) + logger.info('SINGLESTOREDB_URL not set, starting Docker container') + + # Create and start the container + _container_manager = _TestContainerManager() + + if not _container_manager.use_existing: + _container_manager.start() + print(f'Container {_container_manager.container_name} started') + print('Waiting for SingleStoreDB to be ready...') + + # Wait for container to be ready + try: + conn = _container_manager.connect() + conn.close() + print('✓ SingleStoreDB is ready!') + logger.info('Docker container is ready') + except Exception as e: + print(f'✗ Failed to connect to Docker container: {e}') + logger.error(f'Failed to connect to Docker container: {e}') + raise + + # Set the environment variable for all tests + # Check if USE_DATA_API is set to use HTTP connection + if os.environ.get('USE_DATA_API', '0').lower() in ('1', 'true', 'on'): + # Use HTTP URL for tests + url = _container_manager.http_connection_url + if url is None: + raise RuntimeError( + 'Failed to get HTTP URL from container manager', + ) + os.environ['SINGLESTOREDB_URL'] = url + print('=' * 70) + print('USE_DATA_API is enabled - using HTTP Data API for tests') + print(f'Tests will connect via: {url}') + print('=' * 70) + logger.info('USE_DATA_API is enabled - using HTTP Data API for tests') + logger.info(f'Tests will connect via: {url}') + + # Also set INIT_DB_URL to MySQL URL for setup operations + # (like SET GLOBAL) that don't work over HTTP + mysql_url = _container_manager.url + if mysql_url is None: + raise RuntimeError( + 'Failed to get MySQL URL from container manager', + ) + os.environ['SINGLESTOREDB_INIT_DB_URL'] = mysql_url + print(f'Setup operations will use MySQL protocol: {mysql_url}') + logger.info( + f'Setup operations will use MySQL protocol: {mysql_url}', + ) + else: + url = _container_manager.url + if url is None: + raise RuntimeError( + 'Failed to get database URL from container manager', + ) + os.environ['SINGLESTOREDB_URL'] = url + print('=' * 70) + print(f'Tests will connect via MySQL protocol: {url}') + print('=' * 70) + logger.info(f'Tests will connect via MySQL protocol: {url}') + else: + url = os.environ['SINGLESTOREDB_URL'] + logger.debug(f'Using existing SINGLESTOREDB_URL={url}') + + +def pytest_unconfigure(config: pytest.Config) -> None: + """ + Pytest hook that runs after all tests complete. + + Cleans up the Docker container if one was started. + """ + global _container_manager + + if _container_manager is not None and not _container_manager.use_existing: + print('\n' + '=' * 70) + print('Cleaning up Docker container...') + logger.info('Cleaning up Docker container') + try: + _container_manager.stop() + print(f'✓ Container {_container_manager.container_name} stopped') + print('=' * 70) + logger.info('Docker container stopped') + except Exception as e: + print(f'✗ Failed to stop Docker container: {e}') + print('=' * 70) + logger.error(f'Failed to stop Docker container: {e}') + + +@pytest.fixture(scope='session', autouse=True) +def setup_test_environment() -> Iterator[None]: + """ + Automatically set up test environment for all tests. + + This fixture ensures the test environment is ready. The actual container + setup happens in pytest_configure hook to ensure it runs before test + collection. Cleanup happens in pytest_unconfigure hook. + + This fixture exists to ensure proper ordering but doesn't manage the + container lifecycle itself. + """ + # The environment should already be set up by pytest_configure + # This fixture just ensures proper test initialization order + yield + + # Clean up is handled by pytest_unconfigure + + +@pytest.fixture(autouse=True) +def protect_singlestoredb_url() -> Iterator[None]: + """ + Protect SINGLESTOREDB_URL and SINGLESTOREDB_INIT_DB_URL from corruption. + + Some tests (like test_config.py) call reset_option() which resets all + config options to their defaults. Since the 'host' option is registered + with environ=['SINGLESTOREDB_HOST', 'SINGLESTOREDB_URL'], resetting it + overwrites SINGLESTOREDB_URL with just '127.0.0.1' instead of the full + connection string, breaking subsequent tests. + + This fixture saves both URLs before each test and restores them + after, ensuring they're not corrupted. + """ + # Save the current URLs + saved_url = os.environ.get('SINGLESTOREDB_URL') + saved_init_url = os.environ.get('SINGLESTOREDB_INIT_DB_URL') + + yield + + # Restore SINGLESTOREDB_URL if it was set and has been corrupted + if saved_url is not None: + current_url = os.environ.get('SINGLESTOREDB_URL') + if current_url != saved_url: + logger.debug( + f'Restoring SINGLESTOREDB_URL from {current_url!r} to {saved_url!r}', + ) + os.environ['SINGLESTOREDB_URL'] = saved_url + + # Restore SINGLESTOREDB_INIT_DB_URL if it was set and has been corrupted + if saved_init_url is not None: + current_init_url = os.environ.get('SINGLESTOREDB_INIT_DB_URL') + if current_init_url != saved_init_url: + logger.debug( + f'Restoring SINGLESTOREDB_INIT_DB_URL from ' + f'{current_init_url!r} to {saved_init_url!r}', + ) + os.environ['SINGLESTOREDB_INIT_DB_URL'] = saved_init_url diff --git a/singlestoredb/tests/test_plugin.py b/singlestoredb/tests/test_plugin.py index 086b4d82..823cb22b 100644 --- a/singlestoredb/tests/test_plugin.py +++ b/singlestoredb/tests/test_plugin.py @@ -5,10 +5,22 @@ Each of these tests performs the same simple operation which would fail if any other test had been run on the same database. """ +import os + +import pytest + from singlestoredb.connection import Cursor # pytest_plugins = ('singlestoredb.pytest',) +# Skip all tests in this module when using HTTP Data API +# The singlestoredb_tempdb fixture uses 'USE database' which doesn't work with HTTP +pytestmark = pytest.mark.skipif( + 'http://' in os.environ.get('SINGLESTOREDB_URL', '').lower() or + 'https:/' in os.environ.get('SINGLESTOREDB_URL', '').lower(), + reason='Plugin tests require MySQL protocol (USE database not supported via HTTP)', +) + CREATE_TABLE_STATEMENT = 'create table test_dict (a text)' diff --git a/singlestoredb/tests/utils.py b/singlestoredb/tests/utils.py index 15023c51..54cdad69 100644 --- a/singlestoredb/tests/utils.py +++ b/singlestoredb/tests/utils.py @@ -1,6 +1,7 @@ #!/usr/bin/env python # type: ignore """Utilities for testing.""" +import logging import os import uuid from typing import Any @@ -11,6 +12,9 @@ from singlestoredb.connection import build_params +logger = logging.getLogger(__name__) + + def apply_template(content: str, vars: Dict[str, Any]) -> str: for k, v in vars.items(): key = '{{%s}}' % k @@ -46,12 +50,16 @@ def load_sql(sql_file: str) -> str: elif 'SINGLESTOREDB_DATABASE' in os.environ: dbname = os.environ['SINGLESTOREDB_DATBASE'] - # If no database name was specified, use initializer URL if given. - # HTTP can't change databases, so you can't initialize from HTTP - # while also creating a database. + # Use initializer URL if given for setup operations. + # HTTP can't change databases or execute certain commands like SET GLOBAL, + # so we always use the MySQL protocol URL for initialization. args = {'local_infile': True} - if not dbname and 'SINGLESTOREDB_INIT_DB_URL' in os.environ: + if 'SINGLESTOREDB_INIT_DB_URL' in os.environ: args['host'] = os.environ['SINGLESTOREDB_INIT_DB_URL'] + logger.info( + f'load_sql: Using SINGLESTOREDB_INIT_DB_URL for setup: ' + f'{os.environ["SINGLESTOREDB_INIT_DB_URL"]}', + ) http_port = 0 if 'SINGLESTOREDB_URL' in os.environ: