Convert remaining ORM APIs to support 2.0 style

This is kind of a mixed bag of all kinds to help get us
to 1.4 betas.    The documentation stuff is a work in
progress.    Lots of other relatively small changes to
APIs and things.    More commits will follow to continue
improving the documentation and transitioning to the
1.4/2.0 hybrid documentation.  In particular some refinements
to Session usage models so that it can match Engine's
scoping / transactional patterns, and a decision to
start moving away from "subtransactions" completely.

* add select().from_statement() to produce FromStatement in an
  ORM context

* begin referring to select() that has "plugins" for the few edge
  cases where select() will have ORM-only behaviors

* convert dynamic.AppenderQuery to its own object that can use
  select(), though at the moment it uses Query to support legacy
  join calling forms.

* custom query classes for AppenderQuery are replaced by
  do_orm_execute() hooks for custom actions, a separate gerrit
  will document this

* add Session.get() to replace query.get()

* Deprecate session.begin->subtransaction.  propose within the
  test suite a hypothetical recipe for apps that rely on this
  pattern

* introduce Session construction level context manager,
  sessionmaker context manager, rewrite the whole top of the
  session_transaction.rst documentation.   Establish context manager
  patterns for Session that are identical to engine

* ensure same begin_nested() / commit() behavior as engine

* devise all new "join into an external transaction" recipe,
  add test support for it, add rules into Session so it
  just works, write new docs.  need to ensure this doesn't
  break anything

* vastly reduce the verbosity of lots of session docs as
  I dont think people read this stuff and it's difficult
  to keep current in any case

* constructs like case(), with_only_columns() really need
  to move to *columns, add a coercion rule to just change
  these.

* docs need changes everywhere I look.  in_() is not in
  the Core tutorial?  how do people even know about it?
  Remove tons of cruft from Select docs, etc.

* build a system for common ORM options like populate_existing
  and autoflush to populate from execution options.

* others?

Change-Id: Ia4bea0f804250e54d90b3884cf8aab8b66b82ecf

doc/build/changelog/migration_20.rst

@@ -738,6 +738,7 @@ In 2.0, an application that still wishes to use a separate :class:`_schema.Table
 does not want to use Declarative with ``__table__``, can instead use the above
 pattern which basically does the same thing.
 
+.. _migration_20_unify_select:
 
 ORM Query Unified with Core Select
 ==================================

doc/build/core/future.rst

@@ -3,6 +3,29 @@
 SQLAlchemy 2.0 Future (Core)
 ============================
 
+This package includes a relatively small number of transitional elements
+to allow "2.0 mode" to take place within SQLAlchemy 1.4.   The primary
+objects provided here are :class:`_future.Engine` and :class:`_future.Connection`,
+which are both subclasses of the existing :class:`_engine.Engine` and
+:class:`_engine.Connection` objects with essentially a smaller set of
+methods and the removal of "autocommit".
+
+Within the 1.4 series, the "2.0" style of engines and connections is enabled
+by passing the :paramref:`_sa.create_engine.future` flag to
+:func:`_sa.create_engine`::
+
+    from sqlalchemy import create_engine
+    engine = create_engine("postgresql://user:pass@host/dbname", future=True)
+
+Similarly, with the ORM, to enable "future" behavior in the ORM :class:`.Session`,
+pass the :paramref:`_orm.Session.future` parameter either to the
+:class:`.Session` constructor directly, or via the :class:`_orm.sessionmaker`
+class::
+
+    from sqlalchemy.orm import sessionmaker
+
+    Session = sessionmaker(engine, future=True)
+
 .. seealso::
 
     :ref:`migration_20_toplevel` - Introduction to the 2.0 series of SQLAlchemy

doc/build/core/selectable.rst

@@ -80,16 +80,22 @@ elements are themselves :class:`_expression.ColumnElement` subclasses).
 .. autoclass:: Lateral
    :members:
 
+.. autoclass:: ReturnsRows
+   :members:
+   :inherited-members: ClauseElement
+
 .. autoclass:: ScalarSelect
    :members:
 
 .. autoclass:: Select
    :members:
    :inherited-members:  ClauseElement
-   :exclude-members: memoized_attribute, memoized_instancemethod
+   :exclude-members: memoized_attribute, memoized_instancemethod, append_correlation, append_column, append_prefix, append_whereclause, append_having, append_from, append_order_by, append_group_by
+
 
 .. autoclass:: Selectable
    :members:
+   :inherited-members: ClauseElement
 
 .. autoclass:: SelectBase
    :members:

doc/build/core/tutorial.rst

@@ -716,7 +716,112 @@ will ensure that the return type of the expression is handled as boolean::
 
     somecolumn.bool_op('-->')('some value')
 
-.. versionadded:: 1.2.0b3  Added the :meth:`.Operators.bool_op` method.
+
+Commonly Used Operators
+-------------------------
+
+
+Here's a rundown of some of the most common operators used in both the
+Core expression language as well as in the ORM.  Here we see expressions
+that are most commonly present when using the :meth:`_sql.Select.where` method,
+but can be used in other scenarios as well.
+
+A listing of all the column-level operations common to all column-like
+objects is at :class:`.ColumnOperators`.
+
+
+* :meth:`equals <.ColumnOperators.__eq__>`::
+
+    statement.where(users.c.name == 'ed')
+
+* :meth:`not equals <.ColumnOperators.__ne__>`::
+
+    statement.where(users.c.name != 'ed')
+
+* :meth:`LIKE <.ColumnOperators.like>`::
+
+    statement.where(users.c.name.like('%ed%'))
+
+ .. note:: :meth:`.ColumnOperators.like` renders the LIKE operator, which
+    is case insensitive on some backends, and case sensitive
+    on others.  For guaranteed case-insensitive comparisons, use
+    :meth:`.ColumnOperators.ilike`.
+
+* :meth:`ILIKE <.ColumnOperators.ilike>` (case-insensitive LIKE)::
+
+    statement.where(users.c.name.ilike('%ed%'))
+
+ .. note:: most backends don't support ILIKE directly.  For those,
+    the :meth:`.ColumnOperators.ilike` operator renders an expression
+    combining LIKE with the LOWER SQL function applied to each operand.
+
+* :meth:`IN <.ColumnOperators.in_>`::
+
+    statement.where(users.c..name.in_(['ed', 'wendy', 'jack']))
+
+    # works with Select objects too:
+    statement.where.filter(users.c.name.in_(
+        select(users.c.name).where(users.c.name.like('%ed%'))
+    ))
+
+    # use tuple_() for composite (multi-column) queries
+    from sqlalchemy import tuple_
+    statement.where(
+        tuple_(users.c.name, users.c.nickname).\
+        in_([('ed', 'edsnickname'), ('wendy', 'windy')])
+    )
+
+* :meth:`NOT IN <.ColumnOperators.notin_>`::
+
+    statement.where(~users.c.name.in_(['ed', 'wendy', 'jack']))
+
+* :meth:`IS NULL <.ColumnOperators.is_>`::
+
+    statement.where(users.c. == None)
+
+    # alternatively, if pep8/linters are a concern
+    statement.where(users.c.name.is_(None))
+
+* :meth:`IS NOT NULL <.ColumnOperators.isnot>`::
+
+    statement.where(users.c.name != None)
+
+    # alternatively, if pep8/linters are a concern
+    statement.where(users.c.name.isnot(None))
+
+* :func:`AND <.sql.expression.and_>`::
+
+    # use and_()
+    from sqlalchemy import and_
+    statement.where(and_(users.c.name == 'ed', users.c.fullname == 'Ed Jones'))
+
+    # or send multiple expressions to .where()
+    statement.where(users.c.name == 'ed', users.c.fullname == 'Ed Jones')
+
+    # or chain multiple where() calls
+    statement.where(users.c.name == 'ed').where(users.c.fullname == 'Ed Jones')
+
+ .. note::  Make sure you use :func:`.and_` and **not** the
+    Python ``and`` operator!
+
+* :func:`OR <.sql.expression.or_>`::
+
+    from sqlalchemy import or_
+    statement.where(or_(users.c.name == 'ed', users.c.name == 'wendy'))
+
+ .. note::  Make sure you use :func:`.or_` and **not** the
+    Python ``or`` operator!
+
+* :meth:`MATCH <.ColumnOperators.match>`::
+
+    statement.where(users.c.name.match('wendy'))
+
+ .. note::
+
+    :meth:`~.ColumnOperators.match` uses a database-specific ``MATCH``
+    or ``CONTAINS`` function; its behavior will vary by backend and is not
+    available on some backends such as SQLite.
+
 
 Operator Customization
 ----------------------

doc/build/glossary.rst

@@ -9,6 +9,22 @@ Glossary
 .. glossary::
     :sorted:
 
+    1.x style
+    2.0 style
+    1.x-style
+    2.0-style
+        These terms are new in SQLAlchemy 1.4 and refer to the SQLAlchemy 1.4->
+        2.0 transition plan, described at :ref:`migration_20_toplevel`.  The
+        term "1.x style" refers to an API used in the way it's been documented
+        throughout the 1.x series of SQLAlhcemy and earlier (e.g. 1.3, 1.2, etc)
+        and the term "2.0 style" refers to the way an API will look in version
+        2.0.   Version 1.4 implements nearly all of 2.0's API in so-called
+        "transition mode".
+
+        .. seealso::
+
+            :ref:`migration_20_toplevel`
+
     relational
     relational algebra
 
@@ -49,6 +65,25 @@ Glossary
         in terms of one particular table alias or another, based on its position
         within the join expression.
 
+    plugin
+    plugin-specific
+        "plugin-specific" generally indicates a function or method in
+        SQLAlchemy Core which will behave differently when used in an ORM
+        context.
+
+        SQLAlchemy allows Core consrtucts such as :class:`_sql.Select` objects
+        to participate in a "plugin" system, which can inject additional
+        behaviors and features into the object that are not present by default.
+
+        Specifically, the primary "plugin" is the "orm" plugin, which is
+        at the base of the system that the SQLAlchemy ORM makes use of
+        Core constructs in order to compose and execute SQL queries that
+        return ORM results.
+
+        .. seealso::
+
+            :ref:`migration_20_unify_select`
+
     crud
     CRUD
         An acronym meaning "Create, Update, Delete".  The term in SQL refers to the

doc/build/index.rst

@@ -91,7 +91,7 @@ are documented here.  In contrast to the ORM's domain-centric mode of usage, the
   :doc:`Core Event Interfaces <core/events>` |
   :doc:`Creating Custom SQL Constructs <core/compiler>` |
 
-* **SQLAlchemy 2.0 Compatibility:** :doc:`SQLAlchemy 2.0 Future (Core) <core/future>`
+* **SQLAlchemy 2.0 Compatibility:** :ref:`migration_20_toplevel`
 
 Dialect Documentation
 ======================

doc/build/orm/collections.rst

@@ -32,11 +32,12 @@ loading of child items both at load time as well as deletion time.
 Dynamic Relationship Loaders
 ----------------------------
 
-A key feature to enable management of a large collection is the so-called "dynamic"
-relationship.  This is an optional form of :func:`~sqlalchemy.orm.relationship` which
-returns a :class:`~sqlalchemy.orm.query.Query` object in place of a collection
-when accessed. :func:`~sqlalchemy.orm.query.Query.filter` criterion may be
-applied as well as limits and offsets, either explicitly or via array slices::
+A key feature to enable management of a large collection is the so-called
+"dynamic" relationship.  This is an optional form of
+:func:`_orm.relationship` which returns a
+:class:`_orm.AppenderQuery` object in place of a collection
+when accessed. Filtering criterion may be applied as well as limits and
+offsets, either explicitly or via array slices::
 
     class User(Base):
         __tablename__ = 'user'
@@ -52,7 +53,7 @@ applied as well as limits and offsets, either explicitly or via array slices::
     posts = jack.posts[5:20]
 
 The dynamic relationship supports limited write operations, via the
-``append()`` and ``remove()`` methods::
+:meth:`_orm.AppenderQuery.append` and :meth:`_orm.AppenderQuery.remove` methods::
 
     oldpost = jack.posts.filter(Post.headline=='old post').one()
     jack.posts.remove(oldpost)
@@ -78,6 +79,9 @@ function in conjunction with ``lazy='dynamic'``::
 
 Note that eager/lazy loading options cannot be used in conjunction dynamic relationships at this time.
 
+.. autoclass:: sqlalchemy.orm.AppenderQuery
+    :members:
+
 .. note::
 
    The :func:`_orm.dynamic_loader` function is essentially the same

doc/build/orm/persistence_techniques.rst

@@ -2,6 +2,107 @@
 Additional Persistence Techniques
 =================================
 
+.. _session_deleting_from_collections:
+
+Notes on Delete - Deleting Objects Referenced from Collections and Scalar Relationships
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ORM in general never modifies the contents of a collection or scalar
+relationship during the flush process.  This means, if your class has a
+:func:`_orm.relationship` that refers to a collection of objects, or a reference
+to a single object such as many-to-one, the contents of this attribute will
+not be modified when the flush process occurs.  Instead, it is expected
+that the :class:`.Session` would eventually be expired, either through the expire-on-commit behavior of
+:meth:`.Session.commit` or through explicit use of :meth:`.Session.expire`.
+At that point, any referenced object or collection associated with that
+:class:`.Session` will be cleared and will re-load itself upon next access.
+
+A common confusion that arises regarding this behavior involves the use of the
+:meth:`~.Session.delete` method.   When :meth:`.Session.delete` is invoked upon
+an object and the :class:`.Session` is flushed, the row is deleted from the
+database.  Rows that refer to the target row via  foreign key, assuming they
+are tracked using a :func:`_orm.relationship` between the two mapped object types,
+will also see their foreign key attributes UPDATED to null, or if delete
+cascade is set up, the related rows will be deleted as well. However, even
+though rows related to the deleted object might be themselves modified as well,
+**no changes occur to relationship-bound collections or object references on
+the objects** involved in the operation within the scope of the flush
+itself.   This means if the object was a
+member of a related collection, it will still be present on the Python side
+until that collection is expired.  Similarly, if the object were
+referenced via many-to-one or one-to-one from another object, that reference
+will remain present on that object until the object is expired as well.
+
+Below, we illustrate that after an ``Address`` object is marked
+for deletion, it's still present in the collection associated with the
+parent ``User``, even after a flush::
+
+    >>> address = user.addresses[1]
+    >>> session.delete(address)
+    >>> session.flush()
+    >>> address in user.addresses
+    True
+
+When the above session is committed, all attributes are expired.  The next
+access of ``user.addresses`` will re-load the collection, revealing the
+desired state::
+
+    >>> session.commit()
+    >>> address in user.addresses
+    False
+
+There is a recipe for intercepting :meth:`.Session.delete` and invoking this
+expiration automatically; see `ExpireRelationshipOnFKChange <http://www.sqlalchemy.org/trac/wiki/UsageRecipes/ExpireRelationshipOnFKChange>`_ for this.  However, the usual practice of
+deleting items within collections is to forego the usage of
+:meth:`~.Session.delete` directly, and instead use cascade behavior to
+automatically invoke the deletion as a result of removing the object from the
+parent collection.  The ``delete-orphan`` cascade accomplishes this, as
+illustrated in the example below::
+
+    class User(Base):
+        __tablename__ = 'user'
+
+        # ...
+
+        addresses = relationship(
+            "Address", cascade="all, delete-orphan")
+
+    # ...
+
+    del user.addresses[1]
+    session.flush()
+
+Where above, upon removing the ``Address`` object from the ``User.addresses``
+collection, the ``delete-orphan`` cascade has the effect of marking the ``Address``
+object for deletion in the same way as passing it to :meth:`~.Session.delete`.
+
+The ``delete-orphan`` cascade can also be applied to a many-to-one
+or one-to-one relationship, so that when an object is de-associated from its
+parent, it is also automatically marked for deletion.   Using ``delete-orphan``
+cascade on a many-to-one or one-to-one requires an additional flag
+:paramref:`_orm.relationship.single_parent` which invokes an assertion
+that this related object is not to shared with any other parent simultaneously::
+
+    class User(Base):
+        # ...
+
+        preference = relationship(
+            "Preference", cascade="all, delete-orphan",
+            single_parent=True)
+
+
+Above, if a hypothetical ``Preference`` object is removed from a ``User``,
+it will be deleted on flush::
+
+    some_user.preference = None
+    session.flush()  # will delete the Preference object
+
+.. seealso::
+
+    :ref:`unitofwork_cascades` for detail on cascades.
+
+
+
 .. _flush_embedded_sql_expressions:
 
 Embedding SQL Insert/Update Expressions into a Flush

doc/build/orm/session.rst

@@ -6,9 +6,10 @@ Using the Session
 
 .. module:: sqlalchemy.orm.session
 
-The :func:`_orm.mapper` function and :mod:`~sqlalchemy.ext.declarative` extensions
-are the primary configurational interface for the ORM. Once mappings are
-configured, the primary usage interface for persistence operations is the
+The declarative base and ORM mapping functions described at
+:ref:`mapper_config_toplevel` are the primary configurational interface for the
+ORM. Once mappings are configured, the primary usage interface for
+persistence operations is the
 :class:`.Session`.
 
 .. toctree::