diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index de259a902334..27af4f03531e 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -1,15 +1,6 @@
* [Home](index.md)
* [Install](install.md)
* [Docs](docs/index.md)
- * [Tutorial](tutorial/index.md)
- * [Getting Started](tutorial/01-Introduction-to-Ibis.ipynb)
- * [Aggregating and Joining](tutorial/02-Aggregates-Joins.ipynb)
- * [Lazy Mode and Logging](tutorial/03-Expressions-Lazy-Mode-Logging.ipynb)
- * [More Value Expressions](tutorial/04-More-Value-Expressions.ipynb)
- * [Creating and Inserting External Data](tutorial/05-IO-Create-Insert-External-Data.ipynb)
- * [Complex Filtering](tutorial/06-ComplexFiltering.ipynb)
- * [Analytics Tools](tutorial/07-Analytics-Tools.ipynb)
- * [Geospatial Analysis](tutorial/rendered/08-Geospatial-Analysis.ipynb)
* [How To Guide](how_to/)
* [Execution Backends](backends/)
* [User Guide](user_guide/)
diff --git a/docs/docs/index.md b/docs/docs/index.md
index 582dfeed9ab6..7c01364f43e5 100644
--- a/docs/docs/index.md
+++ b/docs/docs/index.md
@@ -4,6 +4,6 @@ Welcome to the ibis documentation!
- **Coming from Pandas?**: Check out [ibis for pandas users](../ibis-for-pandas-users.ipynb)!
- **Coming from SQL?**: Take a look at [ibis for SQL programmers](../ibis-for-sql-programmers.ipynb)!
-- **Want to see some more examples?**: We've got [a set of tutorial notebooks](../tutorial/index.md) for that!
+- **Want to see some more examples?**: We've got [a set of tutorial notebooks](https://github.com/ibis-project/ibis-examples) for that!
- **Looking for API docs?**: Start [here](../api/expressions/top_level.md)!
- **Interested in contributing?**: Our [contribution section](../community/contribute/index.md) has what you need!
diff --git a/docs/index.md b/docs/index.md
index bc0287f55e47..931c4fdab6e7 100644
--- a/docs/index.md
+++ b/docs/index.md
@@ -13,7 +13,7 @@ hide:
---
diff --git a/docs/install.md b/docs/install.md
index ff5e6b838ad9..5439a8e39756 100644
--- a/docs/install.md
+++ b/docs/install.md
@@ -42,5 +42,5 @@ hide:
After you've successfully installed Ibis, try going through the tutorial:
-[Go to the Tutorial](./tutorial/index.md){ .md-button .md-button--primary }
+[Go to the Tutorial](https://github.com/ibis-project/ibis-examples){ .md-button .md-button--primary }
diff --git a/docs/tutorial/01-Introduction-to-Ibis.ipynb b/docs/tutorial/01-Introduction-to-Ibis.ipynb
deleted file mode 100644
index 98452472108e..000000000000
--- a/docs/tutorial/01-Introduction-to-Ibis.ipynb
+++ /dev/null
@@ -1,561 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Getting started\n",
- "\n",
- "To start using Ibis, you need a Python environment with Ibis installed. Follow the installation instructions for SQLite to setup an environment.\n",
- "\n",
- "Once you have your environment ready, to start using Ibis simply import the `ibis` module:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "import ibis"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To make it things easier in this tutorial, we will be using _Ibis interactive mode_. For production code, that will rarely be the case. More details on Ibis non-interactive (aka lazy) mode are covered in the third tutorial, _Expressions, lazy mode and logging queries_.\n",
- "\n",
- "To set the interactive mode, use:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "ibis.options.interactive = True"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next thing we need is to create a connection object. The connection defines where the data is stored and where the computations will be performed.\n",
- "\n",
- "For a comparison to pandas, this is not the same as where the data is imported from (e.g. `pandas.read_sql`). pandas loads data into memory and performs the computations itself. Ibis won't load the data and perform any computation, but instead will leave the data in the backend defined in the connection, and will _ask_ the backend to perform the computations.\n",
- "\n",
- "In this tutorial we will be using a SQLite connection for its simplicity (no installation is needed). But Ibis can work with many different backends, including big data systems, or GPU-accelerated analytical databases. As well as most common relational databases (PostgreSQL, MySQL,...).\n",
- "\n",
- "Let's download the SQLite database from the `ibis-tutorial-data` GCS (Google Cloud Storage) bucket, then connect to it using `ibis`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 16,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "connection = ibis.sqlite.connect(\n",
- " 'geography.db'\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that if you installed Ibis with `pip` instead of `conda`, you may need to install the SQLite backend separately with `pip install 'ibis-framework[sqlite]'`.\n",
- "\n",
- "### Exploring the data\n",
- "\n",
- "To list the tables in the `connection` object, we can use the `.list_tables()` method. If you are using Jupyter, you can see all the methods and attributes of the `connection` object by writing `connection.` and pressing the `` key."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['countries', 'gdp']"
- ]
- },
- "execution_count": 4,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "connection.list_tables()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "These two tables include data about countries, and about GDP by country and year.\n",
- "\n",
- "The data from countries has been obtained from [GeoNames](https://www.geonames.org/countries/).\n",
- "The GDP table will be used in the next tutorial, and the data has been obtained from the\n",
- "[World Bank website](https://data.worldbank.org/indicator/NY.GDP.MKTP.CD).\n",
- "\n",
- "Next, we want to access a specific table in the database. We can create a handler to the `countries` table with:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To list the columns of the `countries` table, we can use the `columns` attribute.\n",
- "\n",
- "Again, Jupyter users can see all the methods and attributes of the `countries` object by typing `countries.` and pressing ``."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['iso_alpha2',\n",
- " 'iso_alpha3',\n",
- " 'iso_numeric',\n",
- " 'fips',\n",
- " 'name',\n",
- " 'capital',\n",
- " 'area_km2',\n",
- " 'population',\n",
- " 'continent']"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries.columns"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can now access a sample of the data. Let's focus on the `name`, `continent` and `population` columns to start with. We can visualize the values of the columns with:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent population\n",
- "0 Andorra EU 84000\n",
- "1 United Arab Emirates AS 4975593\n",
- "2 Afghanistan AS 29121286\n",
- "3 Antigua and Barbuda NA 86754\n",
- "4 Anguilla NA 13254\n",
- ".. ... ... ...\n",
- "247 Yemen AS 23495361\n",
- "248 Mayotte AF 159042\n",
- "249 South Africa AF 49000000\n",
- "250 Zambia AF 13460305\n",
- "251 Zimbabwe AF 13061000\n",
- "\n",
- "[252 rows x 3 columns]"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['name', 'continent', 'population']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The table is too big for all the results to be displayed, and we probably don't want to see all of them at once anyway. For this reason, just the beginning and the end of the results is displayed. Often, the number of rows will be so large that this operation could take a long time.\n",
- "\n",
- "To check how many rows a table has, we can use the `.count()` method:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "252"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries.count()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To fetch just a subset of the rows, we can use the `.limit(n)` method, where `n` is the number of samples we want. In this case we will fetch the first `3` countries from the table:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent population\n",
- "0 Andorra EU 84000\n",
- "1 United Arab Emirates AS 4975593\n",
- "2 Afghanistan AS 29121286"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['name', 'continent', 'population'].limit(3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Filters and order\n",
- "\n",
- "Now that we've got an intuition of the data available in the table `countries`, we will extract some information from it by applying filters and sorting the data.\n",
- "\n",
- "Let's focus on a single continent. We can see a list of unique continents in the table using the `.distinct()` method:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0 EU\n",
- "1 AS\n",
- "2 NA\n",
- "3 AF\n",
- "4 AN\n",
- "5 SA\n",
- "6 OC\n",
- "Name: continent, dtype: object"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries[['continent']].distinct()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We will focus on Asia (`AS` in the table). We can identify which rows belong to Asian countries using the standard Python `==` operator:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0 False\n",
- "1 True\n",
- "2 True\n",
- "3 False\n",
- "4 False\n",
- " ... \n",
- "247 True\n",
- "248 False\n",
- "249 False\n",
- "250 False\n",
- "251 False\n",
- "Name: tmp, Length: 252, dtype: bool"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['continent'] == 'AS'"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The result has a value `True` for rows where the condition is true, and the value `False` when it's not.\n",
- "\n",
- "We can provide this expression to the method `.filter()`, and save the result in the variable `asian_countries` for future use."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {
- "scrolled": true
- },
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent population\n",
- "0 United Arab Emirates AS 4975593\n",
- "1 Afghanistan AS 29121286\n",
- "2 Armenia AS 2968000\n",
- "3 Azerbaijan AS 8303512\n",
- "4 Bangladesh AS 156118464\n",
- "5 Bahrain AS 738004\n",
- "6 Brunei AS 395027\n",
- "7 Bhutan AS 699847\n",
- "8 Cocos [Keeling] Islands AS 628\n",
- "9 China AS 1330044000\n",
- "10 Georgia AS 4630000\n",
- "11 Hong Kong AS 6898686\n",
- "12 Indonesia AS 242968342\n",
- "13 Israel AS 7353985\n",
- "14 India AS 1173108018\n",
- "15 British Indian Ocean Territory AS 4000\n",
- "16 Iraq AS 29671605\n",
- "17 Iran AS 76923300\n",
- "18 Jordan AS 6407085\n",
- "19 Japan AS 127288000\n",
- "20 Kyrgyzstan AS 5776500\n",
- "21 Cambodia AS 14453680\n",
- "22 North Korea AS 22912177\n",
- "23 South Korea AS 48422644\n",
- "24 Kuwait AS 2789132\n",
- "25 Kazakhstan AS 15340000\n",
- "26 Laos AS 6368162\n",
- "27 Lebanon AS 4125247\n",
- "28 Sri Lanka AS 21513990\n",
- "29 Myanmar AS 53414374\n",
- "30 Mongolia AS 3086918\n",
- "31 Macao AS 449198\n",
- "32 Maldives AS 395650\n",
- "33 Malaysia AS 28274729\n",
- "34 Nepal AS 28951852\n",
- "35 Oman AS 2967717\n",
- "36 Philippines AS 99900177\n",
- "37 Pakistan AS 184404791\n",
- "38 Palestine AS 3800000\n",
- "39 Qatar AS 840926\n",
- "40 Saudi Arabia AS 25731776\n",
- "41 Singapore AS 4701069\n",
- "42 Syria AS 22198110\n",
- "43 Thailand AS 67089500\n",
- "44 Tajikistan AS 7487489\n",
- "45 Turkmenistan AS 4940916\n",
- "46 Turkey AS 77804122\n",
- "47 Taiwan AS 22894384\n",
- "48 Uzbekistan AS 27865738\n",
- "49 Vietnam AS 89571130\n",
- "50 Yemen AS 23495361"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "asian_countries = countries['name', 'continent', 'population'].filter(\n",
- " countries['continent'] == 'AS'\n",
- ")\n",
- "asian_countries"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can check how many countries exist in Asia (based on the information in the database) by using the `.count()` method we've already seen:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "51"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "asian_countries.count()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, we want to find the most populated countries in Asia. To obtain them, we are going to sort the countries by the column `population`, and just fetch the first 10. To sort by a column in Ibis, we can use the `.order_by()` method:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent population\n",
- "0 Cocos [Keeling] Islands AS 628\n",
- "1 British Indian Ocean Territory AS 4000\n",
- "2 Brunei AS 395027\n",
- "3 Maldives AS 395650\n",
- "4 Macao AS 449198\n",
- "5 Bhutan AS 699847\n",
- "6 Bahrain AS 738004\n",
- "7 Qatar AS 840926\n",
- "8 Kuwait AS 2789132\n",
- "9 Oman AS 2967717"
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "asian_countries.order_by('population').limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This will return the least populated countries, since `.order_by` will by default order in ascending order (ascending order like in `1, 2, 3, 4`). This behavior is consistent with SQL `ORDER BY`.\n",
- "\n",
- "To order in descending order we can use `ibis.desc()`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent population\n",
- "0 China AS 1330044000\n",
- "1 India AS 1173108018\n",
- "2 Indonesia AS 242968342\n",
- "3 Pakistan AS 184404791\n",
- "4 Bangladesh AS 156118464\n",
- "5 Japan AS 127288000\n",
- "6 Philippines AS 99900177\n",
- "7 Vietnam AS 89571130\n",
- "8 Turkey AS 77804122\n",
- "9 Iran AS 76923300"
- ]
- },
- "execution_count": 15,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "asian_countries.order_by(ibis.desc('population')).limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This is the list of the 10 most populated countries based on the data from [GeoNames](https://www.geonames.org/).\n",
- "\n",
- "To learn more about Ibis, continue to the next tutorial."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/02-Aggregates-Joins.ipynb b/docs/tutorial/02-Aggregates-Joins.ipynb
deleted file mode 100644
index 40227dd2a668..000000000000
--- a/docs/tutorial/02-Aggregates-Joins.ipynb
+++ /dev/null
@@ -1,709 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Aggregating and joining data\n",
- "\n",
- "This is the second introductory tutorial to Ibis. If you are new to Ibis, you may want to start\n",
- "by the first tutorial, _01-Introduction-to-Ibis_.\n",
- "\n",
- "In the first tutorial, we saw how to operate on the data of a table. We will work again with\n",
- "the `countries` table as we did previously."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent area_km2 population\n",
- "0 Andorra EU 468.0 84000\n",
- "1 United Arab Emirates AS 82880.0 4975593\n",
- "2 Afghanistan AS 647500.0 29121286\n",
- "3 Antigua and Barbuda NA 443.0 86754\n",
- "4 Anguilla NA 102.0 13254\n",
- ".. ... ... ... ...\n",
- "247 Yemen AS 527970.0 23495361\n",
- "248 Mayotte AF 374.0 159042\n",
- "249 South Africa AF 1219912.0 49000000\n",
- "250 Zambia AF 752614.0 13460305\n",
- "251 Zimbabwe AF 390580.0 13061000\n",
- "\n",
- "[252 rows x 4 columns]"
- ]
- },
- "execution_count": 2,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "import ibis\n",
- "\n",
- "ibis.options.interactive = True\n",
- "\n",
- "connection = ibis.sqlite.connect(\n",
- " 'geography.db'\n",
- ")\n",
- "countries = connection.table('countries')\n",
- "\n",
- "countries['name', 'continent', 'area_km2', 'population']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Expressions\n",
- "\n",
- "We will continue by exploring the data by continent. We will start by creating an expression\n",
- "with the continent names, since our table only contains the abbreviations.\n",
- "\n",
- "An expression is one or more operations performed over the data. They can be used to retrieve the\n",
- "data or to build more complex operations.\n",
- "\n",
- "In this case we will use a `case` conditional statement to replace values depending on a condition.\n",
- "A `case` expression will return a case builder, and must be followed by one or more `when` calls,\n",
- "optionally an `else_` call, and must end with a call to `end`, to complete the full expression.\n",
- "The expression where `case` is called (`countries['continent']` in this case)\n",
- "is evaluated to see if it's equal to any of the first arguments of the calls to `when`. And the second\n",
- "argument is returned. If the value does not match any of the `when` values, the value of `else_` is returned."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "0 Europe\n",
- "1 Asia\n",
- "2 Asia\n",
- "3 North America\n",
- "4 North America\n",
- " ... \n",
- "247 Asia\n",
- "248 Africa\n",
- "249 Africa\n",
- "250 Africa\n",
- "251 Africa\n",
- "Name: tmp, Length: 252, dtype: object"
- ]
- },
- "execution_count": 2,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "continent_name = (\n",
- " countries['continent']\n",
- " .case()\n",
- " .when('NA', 'North America')\n",
- " .when('SA', 'South America')\n",
- " .when('EU', 'Europe')\n",
- " .when('AF', 'Africa')\n",
- " .when('AS', 'Asia')\n",
- " .when('OC', 'Oceania')\n",
- " .when('AN', 'Antarctica')\n",
- " .else_('Unknown continent')\n",
- " .end()\n",
- " .name('continent_name')\n",
- ")\n",
- "continent_name"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "What we did is take the values of the column `countries['continent']`, and we created a calculated\n",
- "column with the names of the continents, as defined in the `when` methods.\n",
- "\n",
- "This calculated column is an expression. The computations didn't happen when defining the `continent_name`\n",
- "variable, and the results are not stored. They have been computed when we printed its content.\n",
- "\n",
- "We can see that by checking the type of `continent_name`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "ibis.expr.types.StringColumn"
- ]
- },
- "execution_count": 3,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "type(continent_name)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In the next tutorial we will see more about eager and lazy mode, and when operations are being\n",
- "executed. For now we can think that the query to the database happens only when we want to see\n",
- "the results.\n",
- "\n",
- "The important part is that now we can use our `continent_name` expression in other expressions.\n",
- "For example, since this is a column (a `StringColumn` to be specific), we can use it as a column\n",
- "to query the countries table.\n",
- "\n",
- "Note that when we created the expression we added `.name('continent_name')` to it, so the column\n",
- "has a name when being returned."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent_name area_km2 population\n",
- "0 Andorra Europe 468.0 84000\n",
- "1 United Arab Emirates Asia 82880.0 4975593\n",
- "2 Afghanistan Asia 647500.0 29121286\n",
- "3 Antigua and Barbuda North America 443.0 86754\n",
- "4 Anguilla North America 102.0 13254\n",
- ".. ... ... ... ...\n",
- "247 Yemen Asia 527970.0 23495361\n",
- "248 Mayotte Africa 374.0 159042\n",
- "249 South Africa Africa 1219912.0 49000000\n",
- "250 Zambia Africa 752614.0 13460305\n",
- "251 Zimbabwe Africa 390580.0 13061000\n",
- "\n",
- "[252 rows x 4 columns]"
- ]
- },
- "execution_count": 4,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['name', continent_name, 'area_km2', 'population']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Just for illustration, let's repeat the same query, but renaming the expression to `continent`\n",
- "when using it in the list of columns to fetch."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " name continent area_km2 population\n",
- "0 Andorra Europe 468.0 84000\n",
- "1 United Arab Emirates Asia 82880.0 4975593\n",
- "2 Afghanistan Asia 647500.0 29121286\n",
- "3 Antigua and Barbuda North America 443.0 86754\n",
- "4 Anguilla North America 102.0 13254\n",
- ".. ... ... ... ...\n",
- "247 Yemen Asia 527970.0 23495361\n",
- "248 Mayotte Africa 374.0 159042\n",
- "249 South Africa Africa 1219912.0 49000000\n",
- "250 Zambia Africa 752614.0 13460305\n",
- "251 Zimbabwe Africa 390580.0 13061000\n",
- "\n",
- "[252 rows x 4 columns]"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['name', continent_name.name('continent'), 'area_km2', 'population']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Aggregating data\n",
- "\n",
- "Now, let's group our data by continent, and let's find the total population of each."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " continent_name total_population\n",
- "0 Africa 1021238685\n",
- "1 Antarctica 170\n",
- "2 Asia 4130584841\n",
- "3 Europe 750724554\n",
- "4 North America 540204371\n",
- "5 Oceania 36067549\n",
- "6 South America 400143568"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries.group_by(continent_name).aggregate(\n",
- " countries['population'].sum().name('total_population')\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can see how Asia is the most populated country, followed by Africa. Antarctica is the least populated,\n",
- "as we would expect.\n",
- "\n",
- "The code to aggregate has two main parts:\n",
- "- The `group_by` method, that receive the column, expression or list of them to group by\n",
- "- The `aggregate` method, that receives an expression with the reduction we want to apply\n",
- "\n",
- "To make things a bit clearer, let's first save the reduction."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "6878963738"
- ]
- },
- "execution_count": 7,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "total_population = countries['population'].sum().name('total_population')\n",
- "total_population"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As we can see, if we perform the operation directly, we will get the sum of the total in the column.\n",
- "\n",
- "But if we take the `total_population` expression as the parameter of the `aggregate` method, then the total is computed\n",
- "over every group defined by the `group_by` method."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " continent_name total_population\n",
- "0 Africa 1021238685\n",
- "1 Antarctica 170\n",
- "2 Asia 4130584841\n",
- "3 Europe 750724554\n",
- "4 North America 540204371\n",
- "5 Oceania 36067549\n",
- "6 South America 400143568"
- ]
- },
- "execution_count": 8,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries.group_by(continent_name).aggregate(total_population)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If we want to compute two aggregates at the same time, we can pass a list to the `aggregate` method.\n",
- "\n",
- "For illustration, we use the `continent` column, instead of the `continent_names` expression. We can\n",
- "use both column names and expressions, and also a list with any of them (e.g. `[continent_names, 'name']`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " continent total_population average_area\n",
- "0 AF 1021238685 5.234534e+05\n",
- "1 AN 170 2.802439e+06\n",
- "2 AS 4130584841 6.196685e+05\n",
- "3 EU 750724554 4.293017e+05\n",
- "4 NA 540204371 5.836313e+05\n",
- "5 OC 36067549 3.044157e+05\n",
- "6 SA 400143568 1.272751e+06"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries.group_by('continent').aggregate(\n",
- " [total_population, countries['area_km2'].mean().name('average_area')]\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Joining data\n",
- "\n",
- "Now we are going to get the total gross domestic product (GDP) for each continent. In this case, the GDP data\n",
- "is not in the same table `countries`, but in a table `gdp`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " country_code year value\n",
- "0 ABW 1986 4.054634e+08\n",
- "1 ABW 1987 4.876025e+08\n",
- "2 ABW 1988 5.964236e+08\n",
- "3 ABW 1989 6.953044e+08\n",
- "4 ABW 1990 7.648871e+08\n",
- "... ... ... ...\n",
- "9995 SVK 2002 3.513034e+10\n",
- "9996 SVK 2003 4.681659e+10\n",
- "9997 SVK 2004 5.733202e+10\n",
- "9998 SVK 2005 6.278531e+10\n",
- "9999 SVK 2006 7.070810e+10\n",
- "\n",
- "[10000 rows x 3 columns]"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "gdp = connection.table('gdp')\n",
- "gdp"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The table contains information for different years, we can easily check the range with:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "(1960, 2017)"
- ]
- },
- "execution_count": 11,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "gdp['year'].min(), gdp['year'].max()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Now, we are going to join this data with the `countries` table so we can obtain the continent\n",
- "of each country. The `countries` table has several different codes for the countries. Let's find out which\n",
- "one matches the three letter code in the `gdp` table."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " iso_alpha2 iso_alpha3 iso_numeric fips name\n",
- "0 AD AND 20 AN Andorra\n",
- "1 AE ARE 784 AE United Arab Emirates\n",
- "2 AF AFG 4 AF Afghanistan\n",
- "3 AG ATG 28 AC Antigua and Barbuda\n",
- "4 AI AIA 660 AV Anguilla\n",
- ".. ... ... ... ... ...\n",
- "247 YE YEM 887 YM Yemen\n",
- "248 YT MYT 175 MF Mayotte\n",
- "249 ZA ZAF 710 SF South Africa\n",
- "250 ZM ZMB 894 ZA Zambia\n",
- "251 ZW ZWE 716 ZI Zimbabwe\n",
- "\n",
- "[252 rows x 5 columns]"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['iso_alpha2', 'iso_alpha3', 'iso_numeric', 'fips', 'name']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `country_code` in `gdp` corresponds to `iso_alpha3` in the `countries` table. We can also see\n",
- "how the `gdp` table has `10,000` rows, while `countries` has `252`. We will start joining the\n",
- "two tables by the codes that match, discarding the codes that do not exist in both tables.\n",
- "This is called an inner join."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " iso_alpha2 iso_alpha3 iso_numeric fips name capital \\\n",
- "0 AD AND 20 AN Andorra Andorra la Vella \n",
- "1 AD AND 20 AN Andorra Andorra la Vella \n",
- "2 AD AND 20 AN Andorra Andorra la Vella \n",
- "3 AD AND 20 AN Andorra Andorra la Vella \n",
- "4 AD AND 20 AN Andorra Andorra la Vella \n",
- "... ... ... ... ... ... ... \n",
- "9482 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "9483 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "9484 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "9485 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "9486 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "\n",
- " area_km2 population continent country_code year value \n",
- "0 468.0 84000 EU AND 1970 7.861921e+07 \n",
- "1 468.0 84000 EU AND 1971 8.940982e+07 \n",
- "2 468.0 84000 EU AND 1972 1.134082e+08 \n",
- "3 468.0 84000 EU AND 1973 1.508201e+08 \n",
- "4 468.0 84000 EU AND 1974 1.865587e+08 \n",
- "... ... ... ... ... ... ... \n",
- "9482 390580.0 13061000 AF ZWE 2013 1.909102e+10 \n",
- "9483 390580.0 13061000 AF ZWE 2014 1.949552e+10 \n",
- "9484 390580.0 13061000 AF ZWE 2015 1.996312e+10 \n",
- "9485 390580.0 13061000 AF ZWE 2016 2.054868e+10 \n",
- "9486 390580.0 13061000 AF ZWE 2017 2.281301e+10 \n",
- "\n",
- "[9487 rows x 12 columns]"
- ]
- },
- "execution_count": 13,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries_and_gdp = countries.inner_join(\n",
- " gdp, predicates=countries['iso_alpha3'] == gdp['country_code']\n",
- ")\n",
- "countries_and_gdp[countries, gdp]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We joined the table with the information for all years. Now we are going to just take the information about the last available year, 2017."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 14,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " country_code year value\n",
- "0 ABW 2017 2.700559e+09\n",
- "1 AFG 2017 2.019176e+10\n",
- "2 AGO 2017 1.221238e+11\n",
- "3 ALB 2017 1.302506e+10\n",
- "4 AND 2017 3.013387e+09\n",
- ".. ... ... ...\n",
- "242 XKX 2017 7.227700e+09\n",
- "243 YEM 2017 2.681870e+10\n",
- "244 ZAF 2017 3.495541e+11\n",
- "245 ZMB 2017 2.586814e+10\n",
- "246 ZWE 2017 2.281301e+10\n",
- "\n",
- "[247 rows x 3 columns]"
- ]
- },
- "execution_count": 14,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "gdp_2017 = gdp.filter(gdp['year'] == 2017)\n",
- "gdp_2017"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Joining with the new expression we get:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 15,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- " iso_alpha2 iso_alpha3 iso_numeric fips name capital \\\n",
- "0 AW ABW 533 AA Aruba Oranjestad \n",
- "1 AF AFG 4 AF Afghanistan Kabul \n",
- "2 AO AGO 24 AO Angola Luanda \n",
- "3 AL ALB 8 AL Albania Tirana \n",
- "4 AD AND 20 AN Andorra Andorra la Vella \n",
- ".. ... ... ... ... ... ... \n",
- "196 XK XKX 0 KV Kosovo Pristina \n",
- "197 YE YEM 887 YM Yemen Sanaa \n",
- "198 ZA ZAF 710 SF South Africa Pretoria \n",
- "199 ZM ZMB 894 ZA Zambia Lusaka \n",
- "200 ZW ZWE 716 ZI Zimbabwe Harare \n",
- "\n",
- " area_km2 population continent country_code year value \n",
- "0 193.0 71566 NA ABW 2017 2.700559e+09 \n",
- "1 647500.0 29121286 AS AFG 2017 2.019176e+10 \n",
- "2 1246700.0 13068161 AF AGO 2017 1.221238e+11 \n",
- "3 28748.0 2986952 EU ALB 2017 1.302506e+10 \n",
- "4 468.0 84000 EU AND 2017 3.013387e+09 \n",
- ".. ... ... ... ... ... ... \n",
- "196 10908.0 1800000 EU XKX 2017 7.227700e+09 \n",
- "197 527970.0 23495361 AS YEM 2017 2.681870e+10 \n",
- "198 1219912.0 49000000 AF ZAF 2017 3.495541e+11 \n",
- "199 752614.0 13460305 AF ZMB 2017 2.586814e+10 \n",
- "200 390580.0 13061000 AF ZWE 2017 2.281301e+10 \n",
- "\n",
- "[201 rows x 12 columns]"
- ]
- },
- "execution_count": 15,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries_and_gdp = countries.inner_join(\n",
- " gdp_2017, predicates=countries['iso_alpha3'] == gdp_2017['country_code']\n",
- ")\n",
- "countries_and_gdp[countries, gdp_2017]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We have called the `inner_join` method of the `countries` table and passed\n",
- "the `gdp` table as a parameter. The method receives a second parameter, `predicates`, that is used to specify\n",
- "how the join will be performed. In this case we want the `iso_alpha3` column in `countries` to\n",
- "match the `country_code` column in `gdp`. This is specified with the expression\n",
- "`countries['iso_alpha3'] == gdp['country_code']`.\n"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/03-Expressions-Lazy-Mode-Logging.ipynb b/docs/tutorial/03-Expressions-Lazy-Mode-Logging.ipynb
deleted file mode 100644
index 5b875d0bd075..000000000000
--- a/docs/tutorial/03-Expressions-Lazy-Mode-Logging.ipynb
+++ /dev/null
@@ -1,707 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Lazy Mode and Logging\n",
- "\n",
- "So far, we have seen Ibis in interactive mode. Interactive mode (also known as eager mode) makes Ibis return the\n",
- "results of an operation immediately.\n",
- "\n",
- "In most cases, instead of using interactive mode, it makes more sense to use the default lazy mode.\n",
- "In lazy mode, Ibis won't be executing the operations automatically, but instead, will generate an\n",
- "expression to be executed at a later time.\n",
- "\n",
- "Let's see this in practice, starting with the same example as in previous tutorials - the geography database."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "import ibis\n",
- "\n",
- "connection = ibis.sqlite.connect('geography.db')\n",
- "countries = connection.table('countries')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "In previous tutorials, we set interactive mode to `True`, and we obtained the result\n",
- "of every operation."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
┏━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━┓\n",
- "┃ name ┃ continent ┃ population ┃\n",
- "┡━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━┩\n",
- "│ string │ string │ int32 │\n",
- "├──────────────────────┼───────────┼────────────┤\n",
- "│ Andorra │ EU │ 84000 │\n",
- "│ United Arab Emirates │ AS │ 4975593 │\n",
- "│ Afghanistan │ AS │ 29121286 │\n",
- "└──────────────────────┴───────────┴────────────┘\n",
- "
\n"
- ],
- "text/plain": [
- "r0 := AlchemyTable: countries\n",
- " iso_alpha2 string\n",
- " iso_alpha3 string\n",
- " iso_numeric int32\n",
- " fips string\n",
- " name string\n",
- " capital string\n",
- " area_km2 float64\n",
- " population int32\n",
- " continent string\n",
- "\n",
- "r1 := Selection[r0]\n",
- " selections:\n",
- " name: r0.name\n",
- " continent: r0.continent\n",
- " population: r0.population\n",
- "\n",
- "Limit[r1, n=3]"
- ]
- },
- "execution_count": 4,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "ibis.options.interactive = False\n",
- "\n",
- "countries['name', 'continent', 'population'].limit(3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "What we find is the graph of the expressions that would return the desired result instead of the result itself.\n",
- "\n",
- "Let's analyze the expressions in the graph:\n",
- "\n",
- "- We query the `countries` table (all rows and all columns)\n",
- "- We select the `name`, `continent` and `population` columns\n",
- "- We limit the results to only the first `3` rows\n",
- "\n",
- "Now consider that the data is in a database, possibly in a different host than the one executing Ibis.\n",
- "Also consider that the results returned to the user need to be moved to the memory of the host executing Ibis.\n",
- "\n",
- "When using interactive (or eager) mode, if we perform one operation at a time, we would do:\n",
- "\n",
- "- We would move all the rows and columns from the backend (database, big data system, etc.) into memory\n",
- "- Once in memory, we would discard all the columns but `name`, `continent` and `population`\n",
- "- After that, we would discard all the rows, except the first `3`\n",
- "\n",
- "This is not very efficient. If you consider that the table can have millions of rows, backed by a\n",
- "big data system like Spark or Impala, this may not even be possible (not enough memory to load all the data).\n",
- "\n",
- "The solution is to use lazy mode. In lazy mode, instead of obtaining the results after each operation,\n",
- "we build an expression (a graph) of all the operations that need to be done. After all the operations\n",
- "are recorded, the graph is sent to the backend which will perform the operation in an efficient way - only\n",
- "moving to memory the required data.\n",
- "\n",
- "You can think of this as writing a shopping list and requesting someone to go to the supermarket and buy\n",
- "everything you need once the list is complete. As opposed as getting someone to bring all the products of\n",
- "the supermarket to your home and then return everything you don't want.\n",
- "\n",
- "Let's continue with our example, save the expression in a variable `countries_expression`, and check its type."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "ibis.expr.types.relations.Table"
- ]
- },
- "execution_count": 5,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries_expression = countries['name', 'continent', 'population'].limit(3)\n",
- "type(countries_expression)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The type is an Ibis `TableExpr`, since the result is a table (in a broad way, you can consider it a dataframe).\n",
- "\n",
- "Now we have our query instructions (our expression, fetching only 3 columns and 3 rows) in the variable `countries_expression`.\n",
- "\n",
- "At this point, nothing has been requested from the database. We have defined what we want to extract, but we didn't\n",
- "request it from the database yet. We can continue building our expression if we haven't finished yet. Or once we\n",
- "are done, we can simply request it from the database using the method `.execute()`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
name
\n",
- "
continent
\n",
- "
population
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
Andorra
\n",
- "
EU
\n",
- "
84000
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
United Arab Emirates
\n",
- "
AS
\n",
- "
4975593
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
Afghanistan
\n",
- "
AS
\n",
- "
29121286
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " name continent population\n",
- "0 Andorra EU 84000\n",
- "1 United Arab Emirates AS 4975593\n",
- "2 Afghanistan AS 29121286"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries_expression.execute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can build other types of expressions, for example, one that instead of returning a table,\n",
- "returns a columns."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n"
- ],
- "text/plain": [
- "r0 := AlchemyTable: countries\n",
- " iso_alpha2 string\n",
- " iso_alpha3 string\n",
- " iso_numeric int32\n",
- " fips string\n",
- " name string\n",
- " capital string\n",
- " area_km2 float64\n",
- " population int32\n",
- " continent string\n",
- "\n",
- "r1 := Selection[r0]\n",
- " selections:\n",
- " name: r0.name\n",
- " continent: r0.continent\n",
- " population_in_millions: r0.population / 1000000\n",
- "\n",
- "Limit[r1, n=3]"
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "countries['name', 'continent', population_in_millions].limit(3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Since we are in lazy mode (not interactive), those expressions don't request any data from the database\n",
- "unless explicitly requested with `.execute()`.\n",
- "\n",
- "## Logging queries\n",
- "\n",
- "For SQL backends (and for others when it makes sense), the query sent to the database can be logged.\n",
- "This can be done by setting the `verbose` option to `True`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "SELECT t0.name, t0.continent, t0.population / CAST(:param_1 AS REAL) AS population_in_millions \n",
- "FROM main.countries AS t0\n",
- " LIMIT :param_2\n"
- ]
- },
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
name
\n",
- "
continent
\n",
- "
population_in_millions
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
Andorra
\n",
- "
EU
\n",
- "
0.084000
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
United Arab Emirates
\n",
- "
AS
\n",
- "
4.975593
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
Afghanistan
\n",
- "
AS
\n",
- "
29.121286
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " name continent population_in_millions\n",
- "0 Andorra EU 0.084000\n",
- "1 United Arab Emirates AS 4.975593\n",
- "2 Afghanistan AS 29.121286"
- ]
- },
- "execution_count": 10,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "ibis.options.verbose = True\n",
- "\n",
- "countries['name', 'continent', population_in_millions].limit(3).execute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "By default, the logging is done to the terminal, but we can process the query with a custom function.\n",
- "This allows us to save executed queries to a file, save to a database, send them to a web service, etc.\n",
- "\n",
- "For example, to save queries to a file, we can write a custom function that given a query, saves it to a\n",
- "log file."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "from pathlib import Path\n",
- "\n",
- "\n",
- "def log_query_to_file(query: str) -> None:\n",
- " \"\"\"Log queries to `./tutorial_queries.log`.\"\"\"\n",
- " fname = Path() / 'tutorial_queries.log'\n",
- " query = query.replace(\"\\n\", \" \")\n",
- " with fname.open(mode='a') as f:\n",
- " # log on a single line\n",
- " f.write(f\"{query}\\n\")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Then we can set the `verbose_log` option to the custom function, execute one query,\n",
- "wait one second, and execute another query."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "
\n",
- "\n",
- "
\n",
- " \n",
- "
\n",
- "
\n",
- "
name
\n",
- "
continent
\n",
- "
population_in_millions
\n",
- "
\n",
- " \n",
- " \n",
- "
\n",
- "
0
\n",
- "
Andorra
\n",
- "
EU
\n",
- "
0.084000
\n",
- "
\n",
- "
\n",
- "
1
\n",
- "
United Arab Emirates
\n",
- "
AS
\n",
- "
4.975593
\n",
- "
\n",
- "
\n",
- "
2
\n",
- "
Afghanistan
\n",
- "
AS
\n",
- "
29.121286
\n",
- "
\n",
- " \n",
- "
\n",
- "
"
- ],
- "text/plain": [
- " name continent population_in_millions\n",
- "0 Andorra EU 0.084000\n",
- "1 United Arab Emirates AS 4.975593\n",
- "2 Afghanistan AS 29.121286"
- ]
- },
- "execution_count": 12,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "ibis.options.verbose_log = log_query_to_file\n",
- "\n",
- "countries.execute()\n",
- "countries['name', 'continent', population_in_millions].limit(3).execute()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This has created a log file in `$PWD/tutorial_queries.log` where the executed queries have been logged."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 13,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- " 1\tSELECT t0.iso_alpha2, t0.iso_alpha3, t0.iso_numeric, t0.fips, t0.name, t0.capital, t0.area_km2, t0.population, t0.continent FROM main.countries AS t0\n",
- " 2\tSELECT t0.name, t0.continent, t0.population / CAST(:param_1 AS REAL) AS population_in_millions FROM main.countries AS t0 LIMIT :param_2\n",
- " 3\tSELECT t0.iso_alpha2, t0.iso_alpha3, t0.iso_numeric, t0.fips, t0.name, t0.capital, t0.area_km2, t0.population, t0.continent FROM main.countries AS t0\n",
- " 4\tSELECT t0.name, t0.continent, t0.population / CAST(:param_1 AS REAL) AS population_in_millions FROM main.countries AS t0 LIMIT :param_2\n"
- ]
- }
- ],
- "source": [
- "!cat -n $PWD/tutorial_queries.log"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.9"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/04-More-Value-Expressions.ipynb b/docs/tutorial/04-More-Value-Expressions.ipynb
deleted file mode 100644
index 40a43a62d725..000000000000
--- a/docs/tutorial/04-More-Value-Expressions.ipynb
+++ /dev/null
@@ -1,526 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# More Value Expressions\n",
- "Let's walk through some more value expressions."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "import ibis\n",
- "\n",
- "ibis.options.interactive = True\n",
- "\n",
- "connection = ibis.sqlite.connect(\n",
- " 'geography.db'\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Type casting\n",
- "\n",
- "The Ibis type system supports the most common data types used in analytics, including support for nested types like lists, structs, and maps.\n",
- "\n",
- "Type names can be used to cast from one type to another."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')\n",
- "countries"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')\n",
- "countries.population.cast('float').sum()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries.area_km2.cast('int32').sum()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Case / if-then-else expressions\n",
- "\n",
- "\n",
- "We support a number of variants of the SQL-equivalent `CASE` expression, and will add more API functions over time to meet different use cases and enhance the expressiveness of any branching-based value logic."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = (\n",
- " countries.continent.case()\n",
- " .when('AF', 'Africa')\n",
- " .when('AN', 'Antarctica')\n",
- " .when('AS', 'Asia')\n",
- " .when('EU', 'Europe')\n",
- " .when('NA', 'North America')\n",
- " .when('OC', 'Oceania')\n",
- " .when('SA', 'South America')\n",
- " .else_(countries.continent)\n",
- " .end()\n",
- " .name('continent_name')\n",
- ")\n",
- "\n",
- "expr.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "If the `else_` default condition is not provided, any values not matching one of the conditions will be `NULL`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = (\n",
- " countries.continent.case()\n",
- " .when('AF', 'Africa')\n",
- " .when('AS', 'Asia')\n",
- " .when('EU', 'Europe')\n",
- " .when('NA', 'North America')\n",
- " .when('OC', 'Oceania')\n",
- " .when('SA', 'South America')\n",
- " .end()\n",
- " .name('continent_name_with_nulls')\n",
- ")\n",
- "\n",
- "expr.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To test for an arbitrary series of boolean conditions, use the `case` API method and pass any boolean expressions potentially involving columns of the table:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = (\n",
- " ibis.case()\n",
- " .when(countries.population > 25_000_000, 'big')\n",
- " .when(countries.population < 5_000_000, 'small')\n",
- " .else_('medium')\n",
- " .end()\n",
- " .name('size')\n",
- ")\n",
- "\n",
- "countries['name', 'population', expr].limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Simple ternary-cases (like the Python `X if COND else Y`) can be written using the `ifelse` function:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = (countries.continent == 'AS').ifelse('Asia', 'Not Asia').name('is_asia')\n",
- "\n",
- "countries['name', 'continent', expr].limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Set membership\n",
- "\n",
- "\n",
- "The `isin` and `notin` functions are like their pandas counterparts. These can take:\n",
- "\n",
- "- A list of value expressions, either literal values or other column expressions\n",
- "- An array/column expression of some kind"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "is_america = countries.continent.isin(['NA', 'SA'])\n",
- "countries[is_america].continent.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "You can also check for membership in an array. Here is an example of filtering based on the top 3 (ignoring ties) most frequently-occurring values in the `string_col` column of alltypes:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "top_continents = countries.continent.value_counts().limit(3).continent\n",
- "top_continents_filter = countries.continent.isin(top_continents)\n",
- "expr = countries[top_continents_filter]\n",
- "\n",
- "expr.count()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This is a common enough operation that we provide a special analytical filter function `topk`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries.continent.topk(3)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Cool, huh? More on `topk` later."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Null Checking\n",
- "\n",
- "Like their pandas equivalents, the `isnull` and `notnull` functions return True values if the values are null, or non-null, respectively. For example:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = (\n",
- " countries.continent.case()\n",
- " .when('AF', 'Africa')\n",
- " .when('EU', 'Europe')\n",
- " .when('AS', 'Asia')\n",
- " .end()\n",
- " .name('top_continent_name')\n",
- ")\n",
- "\n",
- "expr.isnull().value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Functions like `isnull` can be combined with `case` expressions or functions like `ifelse` to replace null values with some other value. `ifelse` here will use the first value supplied for any `True` value and the second value for any `False` value. Either value can be a scalar or array. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr2 = expr.isnull().ifelse('Other continent', expr).name('continent')\n",
- "expr2.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Distinct-based operations\n",
- "\n",
- "\n",
- "Ibis supports using `distinct` to remove duplicate rows or values on tables or arrays. For example:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries[['continent']].distinct()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This can be combined with `count` to form a reduction metric:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "metric = countries[['continent']].distinct().count().name('num_continents')\n",
- "metric"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## String operations\n",
- "\n",
- "\n",
- "What's supported is pretty basic right now. We intend to support the full gamut of regular expression munging with a nice API, though in some cases some work will be required on SQLite's backend to support everything. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries[['name']].limit(5)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "At the moment, basic substring operations (`substr`, with conveniences `left` and `right`) and Python-like APIs such as `lower` and `upper` (for case normalization) are supported. So you could count first letter occurrences in a string column like so:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "expr = countries.name.lower().left(1).name('first_letter')\n",
- "expr.value_counts().order_by(ibis.desc('first_letter_count')).limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For fuzzy and regex filtering/searching, you can use one of the following\n",
- "\n",
- "- `like`, works as the SQL `LIKE` keyword\n",
- "- `rlike`, like `re.search` or SQL `RLIKE`\n",
- "- `contains`, like `x in str_value` in Python"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries[countries.name.like('%GE%')].name"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries[countries.name.lower().rlike('.*ge.*')].name"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries[countries.name.lower().contains('ge')].name"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Timestamp operations\n",
- "\n",
- "\n",
- "Date and time functionality is relatively limited at present compared with pandas, but we'll get there. The main things we have right now are\n",
- "\n",
- "- Field access (year, month, day, ...)\n",
- "- Timedeltas\n",
- "- Comparisons with fixed timestamps"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "independence = connection.table('independence')\n",
- "\n",
- "independence[\n",
- " independence.independence_date,\n",
- " independence.independence_date.month().name('month'),\n",
- "].limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Somewhat more comprehensively"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "def get_field(f):\n",
- " return getattr(independence.independence_date, f)().name(f)\n",
- "\n",
- "\n",
- "fields = [\n",
- " 'year',\n",
- " 'month',\n",
- " 'day',\n",
- "] # datetime fields can also use: 'hour', 'minute', 'second', 'millisecond'\n",
- "projection = [independence.independence_date] + [get_field(x) for x in fields]\n",
- "independence[projection].limit(10)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "For timestamp arithmetic and comparisons, check out functions in the top level `ibis` namespace. This include things like `day` and `second`, but also the `ibis.timestamp` function:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "independence[\n",
- " independence.independence_date.min(),\n",
- " independence.independence_date.max(),\n",
- " independence.count().name('nrows'),\n",
- "].distinct()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "independence[independence.independence_date > '2000-01-01'].count()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Some backends support adding offsets. For example:\n",
- "\n",
- "```python\n",
- "independence.independence_date + ibis.interval(days=1)\n",
- "ibis.now() - independence.independence_date\n",
- "```"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/05-IO-Create-Insert-External-Data.ipynb b/docs/tutorial/05-IO-Create-Insert-External-Data.ipynb
deleted file mode 100644
index c42ef5cdd988..000000000000
--- a/docs/tutorial/05-IO-Create-Insert-External-Data.ipynb
+++ /dev/null
@@ -1,279 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Creating and Inserting Data"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "from pathlib import Path\n",
- "from urllib.request import urlretrieve\n",
- "\n",
- "tempdir = Path(tempfile.gettempdir())\n",
- "geography_db_file = tempdir / 'geography.db'\n",
- "\n",
- "if geography_db_file.exists():\n",
- " geography_db_file.unlink()\n",
- "\n",
- "_ = urlretrieve(\n",
- " 'https://storage.googleapis.com/ibis-tutorial-data/geography.db',\n",
- " geography_db_file,\n",
- ")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {
- "scrolled": true
- },
- "outputs": [],
- "source": [
- "import ibis\n",
- "\n",
- "ibis.options.interactive = True\n",
- "\n",
- "connection = ibis.sqlite.connect(geography_db_file)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Creating new tables from Ibis expressions\n",
- "\n",
- "\n",
- "Suppose you have an Ibis expression that produces a table:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')\n",
- "\n",
- "continent_name = (\n",
- " countries.continent.case()\n",
- " .when('AF', 'Africa')\n",
- " .when('AN', 'Antarctica')\n",
- " .when('AS', 'Asia')\n",
- " .when('EU', 'Europe')\n",
- " .when('NA', 'North America')\n",
- " .when('OC', 'Oceania')\n",
- " .when('SA', 'South America')\n",
- " .else_(countries.continent)\n",
- " .end()\n",
- " .name('continent_name')\n",
- ")\n",
- "\n",
- "expr = countries[countries.continent, continent_name].distinct()\n",
- "expr"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "To create a table in the database from the results of this expression, use the connection's `create_table` method:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "connection.create_table('continents', expr)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "continents = connection.table('continents')\n",
- "continents"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Tables can be similarly dropped with `drop_table`"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "connection.drop_table('continents')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Creating new tables from in-memory Pandas dataframes\n",
- "\n",
- "Pandas and NumPy are convenient to create test data in memory as a dataframe. This can then be turned into an Ibis expression using `ibis.memtable`."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import pandas as pd\n",
- "import numpy as np\n",
- "\n",
- "\n",
- "def make_students_df(num_records, random_seed=None):\n",
- " rng = np.random.default_rng(random_seed)\n",
- " return pd.DataFrame(\n",
- " {\n",
- " \"firstname\": rng.choice([\"Alice\", \"Bob\", \"Jane\", \"John\"], size=num_records),\n",
- " \"birth_date\": (\n",
- " pd.to_datetime(\"2021-01-01\")\n",
- " + pd.to_timedelta(rng.integers(0, 365, size=num_records), unit=\"D\")\n",
- " ),\n",
- " \"math_grade\": rng.normal(55, 10, size=num_records).clip(0, 100).round(1),\n",
- " }\n",
- " )\n",
- "\n",
- "students_df = make_students_df(21, random_seed=42)\n",
- "students_memtable = ibis.memtable(students_df)\n",
- "students_memtable\n"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "By default `ibis.memtable` uses the `duckdb` in-memory backend to execute queries against the Pandas dataframe data efficiently.\n",
- "\n",
- "We can then materialize it as a physical table for a specific backend if necessary:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "students_db_file = tempdir / \"ibis_tutorial_students.duckdb\"\n",
- "if students_db_file.exists():\n",
- " students_db_file.unlink()\n",
- "\n",
- "connection = ibis.duckdb.connect(students_db_file)\n",
- "connection.create_table('students', students_memtable)\n",
- "students = connection.table('students')\n",
- "students.group_by(students.birth_date.month()).aggregate(\n",
- " count=students.count(),\n",
- " avg_math_grade=students.math_grade.mean(),\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note that NumPy, Pandas and `ibis.memtable` are only suitable to generate data that fits in memory. To generate data larger than memory, we can generate data in chunks and iteratively insert the chunks using `connection.insert(tablename, pandas_dataframe)`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "connection.insert(students.get_name(), make_students_df(10_000, random_seed=43))\n",
- "students.count()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "connection.insert(students.get_name(), make_students_df(10_000, random_seed=44))\n",
- "students.count()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "students.group_by(students.birth_date.month()).aggregate(\n",
- " count=students.count(),\n",
- " avg_math_grade=students.math_grade.mean(),\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Inserting data into existing tables\n",
- "\n",
- "\n",
- "Some backends support inserting data into existing tables from expressions. This can be done using `connection.insert('table_name', expr)`."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": []
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3.10.6 ('base')",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.6"
- },
- "vscode": {
- "interpreter": {
- "hash": "c91744e846ab1fb46a81a92b1fa828c0e6b1381e7e12fd7b2bb300d813000458"
- }
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/06-ComplexFiltering.ipynb b/docs/tutorial/06-ComplexFiltering.ipynb
deleted file mode 100644
index b537317beabc..000000000000
--- a/docs/tutorial/06-ComplexFiltering.ipynb
+++ /dev/null
@@ -1,227 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Complex Filtering\n",
- "\n",
- "The filtering examples we've shown to this point have been pretty simple, either comparisons between columns or fixed values, or set filter functions like `isin` and `notin`. \n",
- "\n",
- "Ibis supports a number of richer analytical filters that can involve one or more of:\n",
- "\n",
- "- Aggregates computed from the same or other tables\n",
- "- Conditional aggregates (in SQL-speak these are similar to \"correlated subqueries\")\n",
- "- \"Existence\" set filters (equivalent to the SQL `EXISTS` and `NOT EXISTS` keywords)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "import ibis\n",
- "\n",
- "ibis.options.interactive = True\n",
- "\n",
- "connection = ibis.sqlite.connect(\n",
- " 'geography.db'\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Using scalar aggregates in filters"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')\n",
- "countries.limit(5)"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We could always compute some aggregate value from the table and use that in another expression, or we can use a data-derived aggregate in the filter. Take the average of a column. For example the average of countries size:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries.area_km2.mean()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "You can use this expression as a substitute for a scalar value in a filter, and the execution engine will combine everything into a single query rather than having to access the database multiple times. For example, we want to filter European countries larger than the average country size in the world. See how most countries in Europe are smaller than the world average:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "cond = countries.area_km2 > countries.area_km2.mean()\n",
- "expr = countries[(countries.continent == 'EU') & cond]\n",
- "expr"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Conditional aggregates\n",
- "\n",
- "\n",
- "Suppose that we wish to filter using an aggregate computed conditional on some other expressions holding true.\n",
- "\n",
- "For example, we want to filter European countries larger than the average country size, but this time of the average in Africa. African countries have an smaller size compared to the world average, and France gets into the list:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "conditional_avg = countries[countries.continent == 'AF'].area_km2.mean()\n",
- "countries[\n",
- " (countries.continent == 'EU') & (countries.area_km2 > conditional_avg)\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## \"Existence\" filters\n",
- "\n",
- "\n",
- "Some filtering involves checking for the existence of a particular value in a column of another table, or amount the results of some value expression. This is common in many-to-many relationships, and can be performed in numerous different ways, but it's nice to be able to express it with a single concise statement and let Ibis compute it optimally.\n",
- "\n",
- "An example could be finding all countries that had **any** year with a higher GDP than 3 trillion US dollars:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "gdp = connection.table('gdp')\n",
- "gdp"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "cond = ((gdp.country_code == countries.iso_alpha3) & (gdp.value > 3e12)).any()\n",
- "\n",
- "countries[cond]['name']"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Note how this is different than a join between `countries` and `gdp`, which would return one row per year. The method `.any()` is equivalent to filtering with a subquery."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Filtering in aggregations\n",
- "\n",
- "\n",
- "Suppose that you want to compute an aggregation with a subset of the data for _only one_ of the metrics / aggregates in question, and the complete data set with the other aggregates. Most aggregation functions are thus equipped with a `where` argument. Let me show it to you in action:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "arctic = countries.name.isin(\n",
- " [\n",
- " 'United States',\n",
- " 'Canada',\n",
- " 'Finland',\n",
- " 'Greenland',\n",
- " 'Iceland',\n",
- " 'Norway',\n",
- " 'Russia',\n",
- " 'Sweden',\n",
- " ]\n",
- ")\n",
- "\n",
- "metrics = [\n",
- " countries.count().name('# countries'),\n",
- " countries.population.sum().name('total population'),\n",
- " countries.population.sum(where=arctic).name('population arctic countries'),\n",
- "]\n",
- "\n",
- "(countries.group_by(countries.continent).aggregate(metrics))"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/07-Analytics-Tools.ipynb b/docs/tutorial/07-Analytics-Tools.ipynb
deleted file mode 100644
index 7ae1dc35eeb0..000000000000
--- a/docs/tutorial/07-Analytics-Tools.ipynb
+++ /dev/null
@@ -1,238 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Analytics Tools"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Setup"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [],
- "source": [
- "!curl -LsS -o geography.db 'https://storage.googleapis.com/ibis-tutorial-data/geography.db'"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "import tempfile\n",
- "\n",
- "import ibis\n",
- "\n",
- "ibis.options.interactive = True\n",
- "\n",
- "connection = ibis.sqlite.connect(\n",
- " 'geography.db'\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Frequency tables\n",
- "\n",
- "Ibis provides the `value_counts` API, just like pandas, for computing a frequency table for a table column or array expression. You might have seen it used already earlier in the tutorial. "
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "countries = connection.table('countries')\n",
- "countries.continent.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "This can be customized, of course:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "freq = countries.group_by(countries.continent).aggregate(\n",
- " [\n",
- " countries.count().name('# countries'),\n",
- " countries.population.sum().name('total population'),\n",
- " ]\n",
- ")\n",
- "freq"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Binning and histograms\n",
- "\n",
- "\n",
- "Numeric array expressions (columns with numeric type and other array expressions) have `bucket` and `histogram` methods which produce different kinds of binning. These produce category values (the computed bins) that can be used in grouping and other analytics.\n",
- "\n",
- "Some backends implement the `.summary()` method, which can be used to see the general distribution of a column.\n",
- "\n",
- "Let's have a look at a few examples."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Alright then, now suppose we want to split the countries up into some buckets of our choosing for their population:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "buckets = [0, 1e6, 1e7, 1e8, 1e9]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `bucket` function creates a bucketed category from the prices:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bucketed = countries.population.bucket(buckets).name('bucket')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's have a look at the value counts:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bucketed.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The buckets we wrote down define 4 buckets numbered 0 through 3. The `NaN` is a pandas `NULL` value (since that's how pandas represents nulls in numeric arrays), so don't worry too much about that. Since the bucketing ends at 100000, we see there are 4122 values that are over 100000. These can be included in the bucketing with `include_over`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bucketed = countries.population.bucket(buckets, include_over=True).name(\n",
- " 'bucket'\n",
- ")\n",
- "bucketed.value_counts()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "The `bucketed` object here is a special **_category_** type"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bucketed.type()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Category values can either have a known or unknown **_cardinality_**. In this case, there's either 4 or 5 buckets based on how we used the `bucket` function.\n",
- "\n",
- "Labels can be assigned to the buckets at any time using the `label` function:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "metadata": {},
- "outputs": [],
- "source": [
- "bucket_counts = bucketed.value_counts()\n",
- "\n",
- "labeled_bucket = bucket_counts.bucket.label(\n",
- " ['< 1M', '> 1M', '> 10M', '> 100M', '> 1B']\n",
- ").name('bucket_name')\n",
- "\n",
- "expr = bucket_counts[labeled_bucket, bucket_counts].order_by('bucket')\n",
- "expr"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Nice, huh?\n",
- "\n",
- "Some backends implement `histogram(num_bins)`, a linear (fixed size bin) equivalent."
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "Python 3 (ipykernel)",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.10.4"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 4
-}
diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md
deleted file mode 100644
index cda38bcde642..000000000000
--- a/docs/tutorial/index.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Introduction to Ibis
-
-Ibis is a Python framework to access data and perform analytical computations from different sources, in a standard way.
-
-In a way, you can think of Ibis as writing SQL in Python, with a focus on analytics, more than simply accessing data. And aside from SQL databases, you can use it with other backends, including big data systems.
-
-Why not simply use SQL instead? SQL is great and widely used. However, SQL has different flavors for different database engines, and SQL is very difficult to maintain when your queries are very complex. Ibis solves both problems by standardizing your code across backends and making it maintainable. Since Ibis is Python, you can structure your code in different files, functions, name variables, write tests, etc.
-
-This tutorial will guide you through Ibis features and provide practical examples. Some knowledge of Python is assumed and knowledge of SQL will be helpful but not required.
-
-Ibis is open source - if anything can be improved in this tutorial, or in Ibis itself, please open an issue in the Ibis GitHub repository or open a pull request with the fix.
diff --git a/docs/tutorial/rendered/08-Geospatial-Analysis.ipynb b/docs/tutorial/rendered/08-Geospatial-Analysis.ipynb
deleted file mode 100644
index 5350de0975b1..000000000000
--- a/docs/tutorial/rendered/08-Geospatial-Analysis.ipynb
+++ /dev/null
@@ -1,653 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "# Geospatial Analysis"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "One of the most popular extensions to PostgreSQL is PostGIS,\n",
- "which adds support for storing geospatial geometries,\n",
- "as well as functionality for reasoning about and performing operations on those geometries.\n",
- "\n",
- "This is a demo showing how to assemble ibis expressions for a PostGIS-enabled database.\n",
- "We will be using a database that has been loaded with an [Open Street Map](https://www.openstreetmap.org/)\n",
- "extract for Southern California.\n",
- "This extract can be found [here](https://download.geofabrik.de/north-america/us/california/socal.html),\n",
- "and loaded into PostGIS using a tool like [ogr2ogr](https://gdal.org/programs/ogr2ogr.html)."
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Preparation\n",
- "\n",
- "We first need to set up a demonstration database and load it with the sample data.\n",
- "If you have Docker installed, you can download and launch a PostGIS database with the following:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 1,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "cd5e5e60822770ddc628adce3f4d26975fa4cfe0325d81925a6fe70b088253cc\n"
- ]
- }
- ],
- "source": [
- "# Launch the postgis container.\n",
- "# This may take a bit of time if it needs to download the image.\n",
- "!docker run -d -p 5432:5432 --name postgis-db -e POSTGRES_PASSWORD=supersecret mdillon/postgis:9.6-alpine"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Next, we download our OSM extract (about 400 MB):"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 2,
- "metadata": {},
- "outputs": [],
- "source": [
- "!wget https://download.geofabrik.de/north-america/us/california/socal-latest.osm.pbf"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Finally, we load it into the database using `ogr2ogr` (this may take some time):"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 3,
- "metadata": {},
- "outputs": [
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "0...10...20...30...40...50...60...70...80...90...100 - done.\n"
- ]
- }
- ],
- "source": [
- "!ogr2ogr -f PostgreSQL PG:\"dbname='postgres' user='postgres' password='supersecret' port=5432 host='localhost'\" -lco OVERWRITE=yes --config PG_USE_COPY YES socal-latest.osm.pbf"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Connecting to the database\n",
- "\n",
- "We first make the relevant imports, and connect to the PostGIS database:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 4,
- "metadata": {},
- "outputs": [],
- "source": [
- "import os\n",
- "\n",
- "import geopandas\n",
- "\n",
- "import ibis\n",
- "\n",
- "%matplotlib inline"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 5,
- "metadata": {},
- "outputs": [],
- "source": [
- "client = ibis.postgres.connect(\n",
- " url='postgres://postgres:supersecret@localhost:5432/postgres'\n",
- ")"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's look at the tables available in the database:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 6,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/plain": [
- "['geography_columns',\n",
- " 'geometry_columns',\n",
- " 'lines',\n",
- " 'multilinestrings',\n",
- " 'multipolygons',\n",
- " 'other_relations',\n",
- " 'points',\n",
- " 'raster_columns',\n",
- " 'raster_overviews',\n",
- " 'spatial_ref_sys']"
- ]
- },
- "execution_count": 6,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "client.list_tables()"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "As you can see, this Open Street Map extract stores its data according to the geometry type.\n",
- "Let's grab references to the polygon and line tables:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 7,
- "metadata": {},
- "outputs": [],
- "source": [
- "polygons = client.table('multipolygons')\n",
- "lines = client.table('lines')"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Querying the data\n",
- "\n",
- "We query the polygons table for shapes with an administrative level of 8,\n",
- "which corresponds to municipalities.\n",
- "\n",
- "We also reproject some of the column names so we don't have a name collision later."
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 8,
- "metadata": {},
- "outputs": [],
- "source": [
- "cities = polygons[polygons.admin_level == '8']\n",
- "\n",
- "cities = cities[\n",
- " cities.name.name('city_name'), cities.wkb_geometry.name('city_geometry')\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "We can assemble a specific query for the city of Los Angeles,\n",
- "and execute it to get the geometry of the city.\n",
- "This will be useful later when reasoning about other geospatial relationships in the LA area:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 9,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "image/svg+xml": [
- ""
- ],
- "text/plain": [
- ""
- ]
- },
- "execution_count": 9,
- "metadata": {},
- "output_type": "execute_result"
- }
- ],
- "source": [
- "los_angeles = cities[cities.city_name == 'Los Angeles']\n",
- "la_city = los_angeles.execute()\n",
- "la_city_geom = la_city.iloc[0].city_geometry\n",
- "la_city_geom"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "Let's also extract freeways from the lines table,\n",
- "which are indicated by the value `'motorway'` in the highway column:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 10,
- "metadata": {},
- "outputs": [],
- "source": [
- "highways = lines[(lines.highway == 'motorway')]\n",
- "highways = highways[\n",
- " highways.name.name('highway_name'),\n",
- " highways.wkb_geometry.name('highway_geometry'),\n",
- "]"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "## Making a spatial join\n",
- "\n",
- "Let's test a spatial join by selecting all the highways that intersect the city of Los Angeles,\n",
- "or one if its neighbors.\n",
- "\n",
- "We begin by assembling an expression for Los Angeles and its neighbors.\n",
- "We consider a city to be a neighbor if it has any point of intersection\n",
- "(by this critereon we also get Los Angeles itself).\n",
- "\n",
- "We can pass in the city geometry that we selected above when making our query by marking it as a literal value in `ibis`:"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 11,
- "metadata": {},
- "outputs": [],
- "source": [
- "la_neighbors_expr = cities[\n",
- " cities.city_geometry.intersects(\n",
- " ibis.literal(la_city_geom, type='multipolygon;4326:geometry')\n",
- " )\n",
- "]"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": 12,
- "metadata": {},
- "outputs": [
- {
- "data": {
- "text/html": [
- "