pep484: schema API

implement strict typing for schema.py

this module has lots of public API, lots of old decisions
and very hard to follow construction sequences in many
cases, and is also where we get a lot of new feature requests,
so strict typing should help keep things clean.

among improvements here, fixed the pool .info getters
and also figured out how to get ColumnCollection and
related to be covariant so that we may set them up
as returning Column or ColumnClause without any conflicts.

DDL was affected, noting that superclasses of DDLElement
(_DDLCompiles, added recently) can now be passed into
"ddl_if" callables; reorganized ddl into ExecutableDDLElement
as a new name for DDLElement and _DDLCompiles renamed to
BaseDDLElement.

setting up strict also located an API use case that
is completely broken, which is connection.execute(some_default)
returns a scalar value.   This case has been deprecated
and new paths have been set up so that connection.scalar()
may be used.  This likely wasn't possible in previous
versions because scalar() would assume a CursorResult.

The scalar() change also impacts Session as we have explicit
support (since someone had reported it as a regression)
for session.execute(Sequence()) to work.  They will get the
same deprecation message (which omits the word "Connection",
just uses ".execute()" and ".scalar()") and they can then
use Session.scalar() as well.  Getting this to type
correctly while still supporting ORM use cases required
some refactoring, and I also set up a keyword only delimeter
for Session.execute() and related as execution_options /
bind_arguments should always be keyword only, applied these
changes to AsyncSession as well.

Additionally simpify Table __init__ now that we are Python
3 only, we can have positional plus explicit kwargs finally.
Simplify Column.__init__ as well again taking advantage
of kw only arguments.

Fill in most/all __init__ methods in sqltypes.py as
the constructor for types is most of the API.   should
likely do this for dialect-specific types as well.

Apply _InfoType for all info attributes as should have been
done originally and update descriptor decorators.

Change-Id: I3f9f8ff3f1c8858471ff4545ac83d68c88107527

doc/build/changelog/unreleased_20/7631.rst

@@ -3,7 +3,8 @@
     :tickets: 7631
 
     Expanded on the "conditional DDL" system implemented by the
-    :class:`_schema.DDLElement` class to be directly available on
+    :class:`_schema.ExecutableDDLElement` class (renamed from
+    :class:`_schema.DDLElement`) to be directly available on
     :class:`_schema.SchemaItem` constructs such as :class:`_schema.Index`,
     :class:`_schema.ForeignKeyConstraint`, etc. such that the conditional logic
     for generating these elements is included within the default DDL emitting

doc/build/changelog/unreleased_20/exec_default.rst

@@ -0,0 +1,10 @@
+.. change::
+    :tags: bug, engine
+
+    Passing a :class:`.DefaultGenerator` object such as a :class:`.Sequence` to
+    the :meth:`.Connection.execute` method is deprecated, as this method is
+    typed as returning a :class:`.CursorResult` object, and not a plain scalar
+    value. The :meth:`.Connection.scalar` method should be used instead, which
+    has been reworked with new internal codepaths to suit invoking a SELECT for
+    default generation objects without going through the
+    :meth:`.Connection.execute` method.

doc/build/core/constraints.rst

@@ -791,6 +791,10 @@ Constraints API
     :members:
     :inherited-members:
 
+.. autoclass:: HasConditionalDDL
+    :members:
+    :inherited-members:
+
 .. autoclass:: PrimaryKeyConstraint
     :members:
     :inherited-members:

doc/build/core/ddl.rst

@@ -49,7 +49,7 @@ Controlling DDL Sequences
 
 The :class:`_schema.DDL` construct introduced previously also has the
 ability to be invoked conditionally based on inspection of the
-database.  This feature is available using the :meth:`.DDLElement.execute_if`
+database.  This feature is available using the :meth:`.ExecutableDDLElement.execute_if`
 method.  For example, if we wanted to create a trigger but only on
 the PostgreSQL backend, we could invoke this as::
 
@@ -85,7 +85,7 @@ the PostgreSQL backend, we could invoke this as::
         trigger.execute_if(dialect='postgresql')
     )
 
-The :paramref:`.DDLElement.execute_if.dialect` keyword also accepts a tuple
+The :paramref:`.ExecutableDDLElement.execute_if.dialect` keyword also accepts a tuple
 of string dialect names::
 
     event.listen(
@@ -99,7 +99,7 @@ of string dialect names::
         trigger.execute_if(dialect=('postgresql', 'mysql'))
     )
 
-The :meth:`.DDLElement.execute_if` method can also work against a callable
+The :meth:`.ExecutableDDLElement.execute_if` method can also work against a callable
 function that will receive the database connection in use.  In the
 example below, we use this to conditionally create a CHECK constraint,
 first looking within the PostgreSQL catalogs to see if it exists:
@@ -151,7 +151,7 @@ Using the built-in DDLElement Classes
 
 The ``sqlalchemy.schema`` package contains SQL expression constructs that
 provide DDL expressions, all of which extend from the common base
-:class:`.DDLElement`. For example, to produce a ``CREATE TABLE`` statement,
+:class:`.ExecutableDDLElement`. For example, to produce a ``CREATE TABLE`` statement,
 one can use the :class:`.CreateTable` construct:
 
 .. sourcecode:: python+sql
@@ -171,13 +171,13 @@ one can use the :class:`.CreateTable` construct:
 Above, the :class:`~sqlalchemy.schema.CreateTable` construct works like any
 other expression construct (such as ``select()``, ``table.insert()``, etc.).
 All of SQLAlchemy's DDL oriented constructs are subclasses of
-the :class:`.DDLElement` base class; this is the base of all the
+the :class:`.ExecutableDDLElement` base class; this is the base of all the
 objects corresponding to CREATE and DROP as well as ALTER,
 not only in SQLAlchemy but in Alembic Migrations as well.
 A full reference of available constructs is in :ref:`schema_api_ddl`.
 
 User-defined DDL constructs may also be created as subclasses of
-:class:`.DDLElement` itself.   The documentation in
+:class:`.ExecutableDDLElement` itself.   The documentation in
 :ref:`sqlalchemy.ext.compiler_toplevel` has several examples of this.
 
 .. _schema_ddl_ddl_if:
@@ -187,7 +187,7 @@ Controlling DDL Generation of Constraints and Indexes
 
 .. versionadded:: 2.0
 
-While the previously mentioned :meth:`.DDLElement.execute_if` method is
+While the previously mentioned :meth:`.ExecutableDDLElement.execute_if` method is
 useful for custom :class:`.DDL` classes which need to invoke conditionally,
 there is also a common need for elements that are typically related to a
 particular :class:`.Table`, namely constraints and indexes, to also be
@@ -196,7 +196,7 @@ that are specific to a particular backend such as PostgreSQL or SQL Server.
 For this use case, the :meth:`.Constraint.ddl_if` and :meth:`.Index.ddl_if`
 methods may be used against constructs such as :class:`.CheckConstraint`,
 :class:`.UniqueConstraint` and :class:`.Index`, accepting the same
-arguments as the :meth:`.DDLElement.execute_if` method in order to control
+arguments as the :meth:`.ExecutableDDLElement.execute_if` method in order to control
 whether or not their DDL will be emitted in terms of their parent
 :class:`.Table` object.  These methods may be used inline when
 creating the definition for a :class:`.Table`
@@ -271,7 +271,7 @@ statement emitted for the index:
 
 The :meth:`.Constraint.ddl_if` and :meth:`.Index.ddl_if` methods create
 an event hook that may be consulted not just at DDL execution time, as is the
-behavior with :meth:`.DDLElement.execute_if`, but also within the SQL compilation
+behavior with :meth:`.ExecutableDDLElement.execute_if`, but also within the SQL compilation
 phase of the :class:`.CreateTable` object, which is responsible for rendering
 the ``CHECK (num > 5)`` DDL inline within the CREATE TABLE statement.
 As such, the event hook that is received by the :meth:`.Constraint.ddl_if.callable_`
@@ -317,67 +317,58 @@ DDL Expression Constructs API
 
 .. autofunction:: sort_tables_and_constraints
 
-.. autoclass:: DDLElement
+.. autoclass:: BaseDDLElement
     :members:
-    :undoc-members:
 
+.. autoattr:: DDLElement
+
+.. autoclass:: ExecutableDDLElement
+    :members:
 
 .. autoclass:: DDL
     :members:
-    :undoc-members:
 
 .. autoclass:: _CreateDropBase
 
 .. autoclass:: CreateTable
     :members:
-    :undoc-members:
 
 
 .. autoclass:: DropTable
     :members:
-    :undoc-members:
 
 
 .. autoclass:: CreateColumn
     :members:
-    :undoc-members:
 
 
 .. autoclass:: CreateSequence
     :members:
-    :undoc-members:
 
 
 .. autoclass:: DropSequence
     :members:
-    :undoc-members:
 
 
 .. autoclass:: CreateIndex
     :members:
-    :undoc-members:
 
 
 .. autoclass:: DropIndex
     :members:
-    :undoc-members:
 
 
 .. autoclass:: AddConstraint
     :members:
-    :undoc-members:
 
 
 .. autoclass:: DropConstraint
     :members:
-    :undoc-members:
 
 
 .. autoclass:: CreateSchema
     :members:
-    :undoc-members:
 
 
 .. autoclass:: DropSchema
     :members:
-    :undoc-members:

doc/build/core/metadata.rst

@@ -558,27 +558,11 @@ Column, Table, MetaData API
 
 .. attribute:: sqlalchemy.schema.BLANK_SCHEMA
 
-    Symbol indicating that a :class:`_schema.Table` or :class:`.Sequence`
-    should have 'None' for its schema, even if the parent
-    :class:`_schema.MetaData` has specified a schema.
-
-    .. seealso::
-
-        :paramref:`_schema.MetaData.schema`
-
-        :paramref:`_schema.Table.schema`
-
-        :paramref:`.Sequence.schema`
-
-    .. versionadded:: 1.0.14
+    Refers to :attr:`.SchemaConst.BLANK_SCHEMA`.
 
 .. attribute:: sqlalchemy.schema.RETAIN_SCHEMA
 
-    Symbol indicating that a :class:`_schema.Table`, :class:`.Sequence`
-    or in some cases a :class:`_schema.ForeignKey` object, in situations
-    where the object is being copied for a :meth:`.Table.to_metadata`
-    operation, should retain the schema name that it already has.
-
+    Refers to :attr:`.SchemaConst.RETAIN_SCHEMA`
 
 
 .. autoclass:: Column
@@ -589,6 +573,8 @@ Column, Table, MetaData API
 .. autoclass:: MetaData
     :members:
 
+.. autoclass:: SchemaConst
+    :members:
 
 .. autoclass:: SchemaItem
     :members:

lib/sqlalchemy/__init__.py

@@ -55,14 +55,17 @@
 from .pool import QueuePool as QueuePool
 from .pool import SingletonThreadPool as SingleonThreadPool
 from .pool import StaticPool as StaticPool
+from .schema import BaseDDLElement as BaseDDLElement
 from .schema import BLANK_SCHEMA as BLANK_SCHEMA
 from .schema import CheckConstraint as CheckConstraint
 from .schema import Column as Column
 from .schema import ColumnDefault as ColumnDefault
 from .schema import Computed as Computed
 from .schema import Constraint as Constraint
 from .schema import DDL as DDL
+from .schema import DDLElement as DDLElement
 from .schema import DefaultClause as DefaultClause
+from .schema import ExecutableDDLElement as ExecutableDDLElement
 from .schema import FetchedValue as FetchedValue
 from .schema import ForeignKey as ForeignKey
 from .schema import ForeignKeyConstraint as ForeignKeyConstraint

lib/sqlalchemy/dialects/postgresql/base.py

@@ -1448,7 +1448,7 @@ def bind_expression(self, bindvalue):
 from ...sql import roles
 from ...sql import sqltypes
 from ...sql import util as sql_util
-from ...sql.ddl import DDLBase
+from ...sql.ddl import InvokeDDLBase
 from ...types import BIGINT
 from ...types import BOOLEAN
 from ...types import CHAR
@@ -2014,7 +2014,7 @@ def drop(self, bind=None, checkfirst=True):
 
         bind._run_ddl_visitor(self.EnumDropper, self, checkfirst=checkfirst)
 
-    class EnumGenerator(DDLBase):
+    class EnumGenerator(InvokeDDLBase):
         def __init__(self, dialect, connection, checkfirst=False, **kwargs):
             super(ENUM.EnumGenerator, self).__init__(connection, **kwargs)
             self.checkfirst = checkfirst
@@ -2035,7 +2035,7 @@ def visit_enum(self, enum):
 
             self.connection.execute(CreateEnumType(enum))
 
-    class EnumDropper(DDLBase):
+    class EnumDropper(InvokeDDLBase):
         def __init__(self, dialect, connection, checkfirst=False, **kwargs):
             super(ENUM.EnumDropper, self).__init__(connection, **kwargs)
             self.checkfirst = checkfirst

lib/sqlalchemy/engine/base.py

@@ -12,7 +12,6 @@
 from typing import Any
 from typing import Callable
 from typing import cast
-from typing import Dict
 from typing import Iterator
 from typing import List
 from typing import Mapping
@@ -65,13 +64,15 @@
     from ..pool import Pool
     from ..pool import PoolProxiedConnection
     from ..sql import Executable
+    from ..sql._typing import _InfoType
     from ..sql.base import SchemaVisitor
     from ..sql.compiler import Compiled
-    from ..sql.ddl import DDLElement
+    from ..sql.ddl import ExecutableDDLElement
     from ..sql.ddl import SchemaDropper
     from ..sql.ddl import SchemaGenerator
     from ..sql.functions import FunctionElement
     from ..sql.schema import ColumnDefault
+    from ..sql.schema import DefaultGenerator
     from ..sql.schema import HasSchemaAttr
     from ..sql.schema import SchemaItem
 
@@ -561,7 +562,7 @@ def _revalidate_connection(self) -> PoolProxiedConnection:
         raise exc.ResourceClosedError("This Connection is closed")
 
     @property
-    def info(self) -> Dict[str, Any]:
+    def info(self) -> _InfoType:
         """Info dictionary associated with the underlying DBAPI connection
         referred to by this :class:`_engine.Connection`, allowing user-defined
         data to be associated with the connection.
@@ -1157,7 +1158,17 @@ def scalar(
          first row returned.
 
         """
-        return self.execute(statement, parameters, execution_options).scalar()
+        distilled_parameters = _distill_params_20(parameters)
+        try:
+            meth = statement._execute_on_scalar
+        except AttributeError as err:
+            raise exc.ObjectNotExecutableError(statement) from err
+        else:
+            return meth(
+                self,
+                distilled_parameters,
+                execution_options or NO_OPTIONS,
+            )
 
     def scalars(
         self,
@@ -1200,7 +1211,7 @@ def execute(
          * :class:`_expression.TextClause` and
            :class:`_expression.TextualSelect`
          * :class:`_schema.DDL` and objects which inherit from
-           :class:`_schema.DDLElement`
+           :class:`_schema.ExecutableDDLElement`
 
         :param parameters: parameters which will be bound into the statement.
          This may be either a dictionary of parameter names to values,
@@ -1244,7 +1255,7 @@ def _execute_function(
 
     def _execute_default(
         self,
-        default: ColumnDefault,
+        default: DefaultGenerator,
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: _ExecuteOptionsParameter,
     ) -> Any:
@@ -1303,7 +1314,7 @@ def _execute_default(
 
     def _execute_ddl(
         self,
-        ddl: DDLElement,
+        ddl: ExecutableDDLElement,
         distilled_parameters: _CoreMultiExecuteParams,
         execution_options: _ExecuteOptionsParameter,
     ) -> CursorResult:

lib/sqlalchemy/engine/default.py

@@ -829,7 +829,8 @@ def _init_ddl(
         execution_options: _ExecuteOptions,
         compiled_ddl: DDLCompiler,
     ) -> ExecutionContext:
-        """Initialize execution context for a DDLElement construct."""
+        """Initialize execution context for an ExecutableDDLElement
+        construct."""
 
         self = cls.__new__(cls)
         self.root_connection = connection

lib/sqlalchemy/engine/mock.py

@@ -28,7 +28,7 @@
     from .interfaces import Dialect
     from .url import URL
     from ..sql.base import Executable
-    from ..sql.ddl import DDLElement
+    from ..sql.ddl import ExecutableDDLElement
     from ..sql.ddl import SchemaDropper
     from ..sql.ddl import SchemaGenerator
     from ..sql.schema import HasSchemaAttr
@@ -101,8 +101,8 @@ def dump(sql, *multiparams, **params):
 
     :param executor: a callable which receives the arguments ``sql``,
      ``*multiparams`` and ``**params``.  The ``sql`` parameter is typically
-     an instance of :class:`.DDLElement`, which can then be compiled into a
-     string using :meth:`.DDLElement.compile`.
+     an instance of :class:`.ExecutableDDLElement`, which can then be compiled
+     into a string using :meth:`.ExecutableDDLElement.compile`.
 
     .. versionadded:: 1.4 - the :func:`.create_mock_engine` function replaces
        the previous "mock" engine strategy used with