DOC: Document core parts of ibis #1351

cpcloud · 2018-02-10T00:17:24Z

No description provided.

jreback · 2018-02-10T00:31:04Z

docs/source/design.rst

+The main user-facing component of ibis is expressions. The base class of all
+expressions in ibis is the :class:`~ibis.expr.types.Expr` class.
+
+Expressions provide the user facing API, defined in ``ibis/expr/api.py``


you could actually put in the links to code

yep, will do

cpcloud · 2018-02-11T01:51:22Z

@kszucs @wesm @jreback please review when you get a chance

kszucs

Cool! If I recall correctly I've tried to click on Ibis design internals three times :)

jreback · 2018-02-11T15:31:13Z

README.md

- [SQLite](http://sqlite.org/)
- [Pandas DataFrames](https://pandas.pydata.org/) (Experimental)
+- [SQLite](https://www.sqlite.org/)
+- [Pandas](https://pandas.pydata.org/) [DataFrames](http://pandas.pydata.org/pandas-docs/stable/dsintro.html#dataframe) (Experimental)


are you marking this experimental?

Yeah, it hasn't been around that long.

jreback · 2018-02-11T15:32:09Z

docs/source/design.rst


-More to come here.
+Primary Goals
+*************


I think this should be '----' otherwise you have too many levels of nesting (compared to other sections)

jreback · 2018-02-11T15:32:34Z

docs/source/design.rst

+#. Composability
+#. Familiarty
+
+Flow of execution


maybe should capitalize all works in sections (or not), just do this consistently

jreback · 2018-02-11T15:33:23Z

docs/source/design.rst

+   (EXAMPLE)
+#. Some optimizations happen at compile time? (EXAMPLE)
+#. Expressions are compiled
+#. The SQL string that generated by the compiler is sent to the database and


this is true for SQL but not for other backends where the execution can happen locally / remote (e.g. pandas / spark / file based)

Added a note

jreback · 2018-02-11T15:33:44Z

docs/source/design.rst

+#. The database returns some data that is then turned into a pandas DataFrame
+   by ibis
+
+


you might want to have reference tags to each sub-section

jreback · 2018-02-11T15:39:14Z

docs/source/design.rst

+expressions.
+
+The compiler works by translating the different pieces of SQL expression into a
+string or SQLAlchemy expression.


ref to SQLAlchemy docs

jreback · 2018-02-11T15:39:51Z

docs/source/design.rst

+:class:`~ibis.impala.compiler.ImpalaExprTranslator` is one of the subclasses
+that will perform this translation.
+
+Execution


maybe have another section on 'local execution' (pandas / spark / file based)

jreback · 2018-02-11T15:40:18Z

docs/source/extending.rst

+Adding a new operation (``Node`` subclass)
+------------------------------------------
+
+Let's go through adding a `sha1`_ method to ibis, implemented in the BigQuery


ref to BQ (maybe doc-section in ibis)

I'll link to the sha1 wikipedia article here https://en.wikipedia.org/wiki/SHA-1

jreback · 2018-02-11T15:40:53Z

docs/source/extending.rst

+function in BigQuery takes a string or bytes and returns a bytestring of length
+20.
+
+.. code-block:: python


you could do these as ipython blocks (I think) esp if you want to use them later

I'm actually going to move this into a notebook

jreback · 2018-02-11T15:42:16Z

ibis/pandas/core.py

+           execute the arguments of the current node
+
+       execute the current node with its executed arguments
+


I think this rather belongs in the docs proper

So, I put this here so that I could start a trend of adding module level docs for each backend, rather than have one rst file for each backend which will tend to get out of date less often than the module docs.

wesm · 2018-02-12T05:46:22Z

Will endeavor to review tomorrow

wesm

Minor comments, but looks like a great start!

wesm · 2018-02-12T21:49:07Z

docs/source/design.rst

+#. Type safety
+#. Expressiveness
+#. Composability
+#. Familiarty


wesm · 2018-02-12T21:49:24Z

docs/source/design.rst

+#. Expressions are type checked as you create them
+#. Some expressions have some optimizations that happen as the user builds them
+   (EXAMPLE)
+#. Some optimizations happen at compile time? (EXAMPLE)


Backend-specific rewrites

wesm · 2018-02-12T21:50:05Z

docs/source/design.rst

+#. The SQL string that generated by the compiler is sent to the database and
+   executed (this step is skipped for the Pandas backend)
+#. The database returns some data that is then turned into a pandas DataFrame
+   by ibis


We should decide on Ibis vs ibis in prose and be consistent (pandas is of course styled as lowercase)

I'll arbitrarily decide to follow the pandas convention.

wesm · 2018-02-12T21:50:46Z

docs/source/design.rst

+
+Here's an example of each type of expression:
+
+.. code-block:: python


Could do IPython directive (with graphviz disabled)

This won't show them anyways, since I'm not repring the expression, just the type.

wesm · 2018-02-12T21:53:23Z

docs/source/design.rst

+Separation of the :class:`~ibis.expr.types.Node` and
+:class:`~ibis.expr.types.Expr` classes also allows the API to be tied to the
+physical type of the expression rather than the particular operation, making it
+easy to define the API in terms of types rather than specific operations.


Another key point here is that the operator output type will often depend on the input type(s). So the "user API" for the result of an operation is a strongly-typed expression having only the behavior of the actual output type of the operator.

Add strings -> get strings, do things with strings

Add numbers -> get numbers, do things with numbers

Cool, added a blurb about this.

wesm · 2018-02-12T21:54:19Z

docs/source/design.rst

+the instance of the :class:`~ibis.sql.compiler.ExprTranslator` subclass
+specific to the backend being compiled. For example, the
+:class:`~ibis.impala.compiler.ImpalaExprTranslator` is one of the subclasses
+that will perform this translation.


May be worth noting explicitly that SQL is only one target, however the library was designed with first-class SQL support in mind, being the lingua franca of analytics.

cpcloud · 2018-02-13T16:51:29Z

Merging on green

jreback reviewed Feb 10, 2018

View reviewed changes

cpcloud added this to the 0.13 milestone Feb 10, 2018

cpcloud added docs Documentation related issues or PRs feature Features or general enhancements internals Issues or PRs related to ibis's internal APIs labels Feb 10, 2018

kszucs approved these changes Feb 11, 2018

View reviewed changes

jreback reviewed Feb 11, 2018

View reviewed changes

cpcloud force-pushed the more-docs branch from c5307e7 to ce89efc Compare February 11, 2018 18:38

cpcloud force-pushed the more-docs branch 2 times, most recently from e406325 to ae5d981 Compare February 12, 2018 14:26

wesm reviewed Feb 12, 2018

View reviewed changes

cpcloud force-pushed the more-docs branch from ae5d981 to c64b5ea Compare February 13, 2018 16:50

DOC: Design docs

10d07b1

cpcloud force-pushed the more-docs branch from c64b5ea to 10d07b1 Compare February 13, 2018 17:32

cpcloud closed this in 5952b1a Feb 13, 2018

cpcloud deleted the more-docs branch February 13, 2018 19:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DOC: Document core parts of ibis #1351

DOC: Document core parts of ibis #1351

cpcloud commented Feb 10, 2018

jreback Feb 10, 2018

cpcloud Feb 11, 2018

cpcloud commented Feb 11, 2018

kszucs left a comment

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

jreback Feb 11, 2018

jreback Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

jreback Feb 11, 2018

cpcloud Feb 11, 2018

wesm commented Feb 12, 2018

wesm left a comment

wesm Feb 12, 2018

cpcloud Feb 13, 2018

wesm Feb 12, 2018

wesm Feb 12, 2018

cpcloud Feb 13, 2018

wesm Feb 12, 2018

cpcloud Feb 13, 2018

wesm Feb 12, 2018

cpcloud Feb 13, 2018

wesm Feb 12, 2018

cpcloud commented Feb 13, 2018

		#. The database returns some data that is then turned into a pandas DataFrame
		by ibis

		execute the arguments of the current node

		execute the current node with its executed arguments


		Here's an example of each type of expression:

		.. code-block:: python

DOC: Document core parts of ibis #1351

DOC: Document core parts of ibis #1351

Conversation

cpcloud commented Feb 10, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Feb 11, 2018

kszucs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wesm commented Feb 12, 2018

wesm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cpcloud commented Feb 13, 2018