Skip to content
Permalink
Browse files
feat: Add geography support (#228)
  • Loading branch information
jimfulton committed Aug 17, 2021
1 parent 62b2975 commit da7a40309de6ca8063d6dcf6678de96a463344e6
@@ -1,5 +1,5 @@
Alembic support
---------------
^^^^^^^^^^^^^^^

`Alembic <https://alembic.sqlalchemy.org>`_ is a lightweight database
migration tool for usage with the SQLAlchemy Database Toolkit for
@@ -0,0 +1,87 @@
Working with Geographic data
^^^^^^^^^^^^^^^^^^^^^^^^^^^^

BigQuery provides a `GEOGRAPHY data type
<https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#geography_type>`_
for `working with geographic data
<https://cloud.google.com/bigquery/docs/gis-data>`_, including:

- Points,
- Linestrings,
- Polygons, and
- Collections of points, linestrings, and polygons.

Geographic data uses the `WGS84
<https://earth-info.nga.mil/#tab_wgs84-data>`_ coordinate system.

To define a geography column, use the `GEOGRAPHY` data type imported
from the `sqlalchemy_bigquery` module:

.. literalinclude:: samples/snippets/geography.py
:language: python
:dedent: 4
:start-after: [START bigquery_sqlalchemy_create_table_with_geography]
:end-before: [END bigquery_sqlalchemy_create_table_with_geography]

BigQuery has a variety of `SQL geographic functions
<https://cloud.google.com/bigquery/docs/reference/standard-sql/geography_functions>`_
for working with geographic data. Among these are functions for
converting between SQL geometry objects and `standard text (WKT) and
binary (WKB) representations
<https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry>`_.

Geography data is typically represented in Python as text strings in
WKT format or as `WKB` objects, which contain binary data in WKB
format. Querying geographic data returns `WKB` objects and `WKB`
objects may be used in queries. When
calling spatial functions that expect geographic arguments, text
arguments are automatically coerced to geography.

Inserting data
~~~~~~~~~~~~~~

When inserting geography data, you can pass WKT strings, `WKT` objects,
or `WKB` objects:

.. literalinclude:: samples/snippets/geography.py
:language: python
:dedent: 4
:start-after: [START bigquery_sqlalchemy_insert_geography]
:end-before: [END bigquery_sqlalchemy_insert_geography]

Note that in the `lake3` example, we got a `WKB` object by creating a
`WKT` object and getting its `wkb` property. Normally, we'd get `WKB`
objects as results of previous queries.

Queries
~~~~~~~

When performing spacial queries, and geography objects are expected,
you can to pass `WKB` or `WKT` objects:

.. literalinclude:: samples/snippets/geography.py
:language: python
:dedent: 4
:start-after: [START bigquery_sqlalchemy_query_geography_wkb]
:end-before: [END bigquery_sqlalchemy_query_geography_wkb]

In this example, we passed the `geog` attribute of `lake2`, which is a WKB object.

Or you can pass strings in WKT format:

.. literalinclude:: samples/snippets/geography.py
:language: python
:dedent: 4
:start-after: [START bigquery_sqlalchemy_query_geography_text]
:end-before: [END bigquery_sqlalchemy_query_geography_text]

Installing geography support
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

To get geography support, you need to install `sqlalchemy-bigquery`
with the `geography` extra, or separately install `GeoAlchemy2` and
`shapely`.

.. code-block:: console

pip install 'sqlalchemy-bigquery[geography]'
@@ -3,7 +3,9 @@
:maxdepth: 2

README
geography
alembic
reference

Changelog
---------
@@ -0,0 +1,12 @@
API Reference
^^^^^^^^^^^^^

Geography
~~~~~~~~~

.. autoclass:: sqlalchemy_bigquery.geography.GEOGRAPHY
:exclude-members: bind_expression, ElementType, bind_processor

.. automodule:: sqlalchemy_bigquery.geography
:members: WKB, WKT
:exclude-members: GEOGRAPHY
@@ -82,22 +82,6 @@ def lint_setup_py(session):
session.run("python", "setup.py", "check", "--restructuredtext", "--strict")


def install_alembic_for_python_38(session, constraints_path):
"""
install alembic for Python 3.8 unit and system tests
We do not require alembic and most tests should run without it, however
- We run some unit tests (Python 3.8) to cover the alembic
registration that happens when alembic is installed.
- We have a system test that demonstrates working with alembic and
proves that the things we think should work do work. :)
"""
if session.python == "3.8":
session.install("alembic", "-c", constraints_path)


def default(session):
# Install all test dependencies, then install this package in-place.

@@ -114,8 +98,13 @@ def default(session):
constraints_path,
)

install_alembic_for_python_38(session, constraints_path)
session.install("-e", ".", "-c", constraints_path)
if session.python == "3.8":
extras = "[alembic]"
elif session.python == "3.9":
extras = "[geography]"
else:
extras = ""
session.install("-e", f".{extras}", "-c", constraints_path)

# Run py.test against the unit tests.
session.run(
@@ -167,8 +156,13 @@ def system(session):
# Install all test dependencies, then install this package into the
# virtualenv's dist-packages.
session.install("mock", "pytest", "google-cloud-testutils", "-c", constraints_path)
install_alembic_for_python_38(session, constraints_path)
session.install("-e", ".", "-c", constraints_path)
if session.python == "3.8":
extras = "[alembic]"
elif session.python == "3.9":
extras = "[geography]"
else:
extras = ""
session.install("-e", f".{extras}", "-c", constraints_path)

# Run py.test against the system tests.
if system_test_exists:
@@ -216,7 +210,13 @@ def compliance(session):
"-c",
constraints_path,
)
session.install("-e", ".", "-c", constraints_path)
if session.python == "3.8":
extras = "[alembic]"
elif session.python == "3.9":
extras = "[geography]"
else:
extras = ""
session.install("-e", f".{extras}", "-c", constraints_path)

session.run(
"py.test",
@@ -251,7 +251,9 @@ def docs(session):
"""Build the docs for this library."""

session.install("-e", ".")
session.install("sphinx==4.0.1", "alabaster", "recommonmark")
session.install(
"sphinx==4.0.1", "alabaster", "geoalchemy2", "shapely", "recommonmark"
)

shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
session.run(
@@ -274,7 +276,12 @@ def docfx(session):

session.install("-e", ".")
session.install(
"sphinx==4.0.1", "alabaster", "recommonmark", "gcp-sphinx-docfx-yaml"
"sphinx==4.0.1",
"alabaster",
"geoalchemy2",
"shapely",
"recommonmark",
"gcp-sphinx-docfx-yaml",
)

shutil.rmtree(os.path.join("docs", "_build"), ignore_errors=True)
@@ -18,7 +18,7 @@

import synthtool as s
from synthtool import gcp

from synthtool.languages import python

REPO_ROOT = pathlib.Path(__file__).parent.absolute()

@@ -27,10 +27,19 @@
# ----------------------------------------------------------------------------
# Add templated files
# ----------------------------------------------------------------------------
extras = []
extras_by_python = {
"3.8": ["alembic"],
"3.9": ["geography"],
}
templated_files = common.py_library(
unit_test_python_versions=["3.6", "3.7", "3.8", "3.9"],
system_test_python_versions=["3.8", "3.9"],
cov_level=100
cov_level=100,
unit_test_extras=extras,
unit_test_extras_by_python=extras_by_python,
system_test_extras=extras,
system_test_extras_by_python=extras_by_python,
)
s.move(templated_files, excludes=[
# sqlalchemy-bigquery was originally licensed MIT
@@ -77,37 +86,6 @@ def place_before(path, text, *before_text, escape=None):
"nox.options.stop_on_first_error = True",
)

install_alembic_for_python_38 = '''
def install_alembic_for_python_38(session, constraints_path):
"""
install alembic for Python 3.8 unit and system tests
We do not require alembic and most tests should run without it, however
- We run some unit tests (Python 3.8) to cover the alembic
registration that happens when alembic is installed.
- We have a system test that demonstrates working with alembic and
proves that the things we think should work do work. :)
"""
if session.python == "3.8":
session.install("alembic", "-c", constraints_path)
'''

place_before(
"noxfile.py",
"def default",
install_alembic_for_python_38,
)

place_before(
"noxfile.py",
' session.install("-e", ".", ',
" install_alembic_for_python_38(session, constraints_path)",
escape='(')

old_sessions = '''
"unit",
"system",
@@ -125,6 +103,9 @@ def install_alembic_for_python_38(session, constraints_path):

s.replace( ["noxfile.py"], old_sessions, new_sessions)

# Maybe we can get rid of this when we don't need pytest-rerunfailures,
# which we won't need when BQ retries itself:
# https://github.com/googleapis/python-bigquery/pull/837
compliance = '''
@nox.session(python=SYSTEM_TEST_PYTHON_VERSIONS)
def compliance(session):
@@ -153,7 +134,13 @@ def compliance(session):
"-c",
constraints_path,
)
session.install("-e", ".", "-c", constraints_path)
if session.python == "3.8":
extras = "[alembic]"
elif session.python == "3.9":
extras = "[geography]"
else:
extras = ""
session.install("-e", f".{extras}", "-c", constraints_path)
session.run(
"py.test",
@@ -180,6 +167,7 @@ def compliance(session):
escape="()",
)

s.replace(["noxfile.py"], '"alabaster"', '"alabaster", "geoalchemy2", "shapely"')



@@ -201,6 +189,12 @@ def compliance(session):
"""
)

# ----------------------------------------------------------------------------
# Samples templates
# ----------------------------------------------------------------------------

python.py_samples(skip_readmes=True)

# ----------------------------------------------------------------------------
# Final cleanup
# ----------------------------------------------------------------------------
Empty file.
Empty file.
@@ -0,0 +1,20 @@
# Copyright (c) 2021 The sqlalchemy-bigquery Authors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

__version__ = "1.0.0-a1"
@@ -0,0 +1,48 @@
# Copyright (c) 2021 The sqlalchemy-bigquery Authors
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in
# the Software without restriction, including without limitation the rights to
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
# the Software, and to permit persons to whom the Software is furnished to do so,
# subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
# COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
# IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
# CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
SQLAlchemy dialect for Google BigQuery
"""

from google.cloud import bigquery
import pytest
import sqlalchemy
import test_utils.prefixer

prefixer = test_utils.prefixer.Prefixer("python-bigquery-sqlalchemy", "tests/system")


@pytest.fixture(scope="session")
def client():
return bigquery.Client()


@pytest.fixture(scope="session")
def dataset_id(client: bigquery.Client):
project_id = client.project
dataset_id = prefixer.create_prefix()
dataset = bigquery.Dataset(f"{project_id}.{dataset_id}")
dataset = client.create_dataset(dataset)
yield dataset_id
client.delete_dataset(dataset_id, delete_contents=True)


@pytest.fixture(scope="session")
def engine(dataset_id):
return sqlalchemy.create_engine(f"bigquery:///{dataset_id}")

0 comments on commit da7a403

Please sign in to comment.