feat(route): add route for DuckDB news #16183
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
Route
Auto: Route Test Complete
|
Successfully generated as following: http://localhost:1200/duckdb/news - Success ✔️<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>DuckDB News</title>
<link>https://duckdb.org/news/</link>
<atom:link href="http://localhost:1200/duckdb/news" rel="self" type="application/rss+xml"></atom:link>
<description>DuckDB News - Made with love by RSSHub(https://github.com/DIYgod/RSSHub)</description>
<generator>RSSHub</generator>
<webMaster>i@diygod.me (DIYgod)</webMaster>
<language>en</language>
<lastBuildDate>Tue, 16 Jul 2024 12:56:18 GMT</lastBuildDate>
<ttl>5</ttl>
<item>
<title>DuckCon #5 in Seattle</title>
<description></description>
<link>https://duckdb.org/2024/08/15/duckcon5.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/08/15/duckcon5.html</guid>
<pubDate>Thu, 15 Aug 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Memory Management in DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/07/09/memory-management.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/07/09/memory-management.html</guid>
<pubDate>Tue, 09 Jul 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB Community Extensions</title>
<description></description>
<link>https://duckdb.org/2024/07/05/community-extensions.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/07/05/community-extensions.html</guid>
<pubDate>Fri, 05 Jul 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Benchmarking Ourselves over Time at DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/06/26/benchmarks-over-time.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/06/26/benchmarks-over-time.html</guid>
<pubDate>Wed, 26 Jun 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>20 000 Stars on GitHub</title>
<description></description>
<link>https://duckdb.org/2024/06/22/github-20k-stars.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/06/22/github-20k-stars.html</guid>
<pubDate>Sat, 22 Jun 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Command Line Data Processing: Using DuckDB as a Unix Tool</title>
<description></description>
<link>https://duckdb.org/2024/06/20/cli-data-processing-using-duckdb-as-a-unix-tool.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/06/20/cli-data-processing-using-duckdb-as-a-unix-tool.html</guid>
<pubDate>Thu, 20 Jun 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Native Delta Lake Support in DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/06/10/delta.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/06/10/delta.html</guid>
<pubDate>Mon, 10 Jun 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 1.0.0</title>
<description></description>
<link>https://duckdb.org/2024/06/03/announcing-duckdb-100.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/06/03/announcing-duckdb-100.html</guid>
<pubDate>Mon, 03 Jun 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Analyzing Railway Traffic in the Netherlands</title>
<description></description>
<link>https://duckdb.org/2024/05/31/analyzing-railway-traffic-in-the-netherlands.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/05/31/analyzing-railway-traffic-in-the-netherlands.html</guid>
<pubDate>Fri, 31 May 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Access 150k+ Datasets from Hugging Face with DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/05/29/access-150k-plus-datasets-from-hugging-face-with-duckdb.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/05/29/access-150k-plus-datasets-from-hugging-face-with-duckdb.html</guid>
<pubDate>Wed, 29 May 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Vector Similarity Search in DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/05/03/vector-similarity-search-vss.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/05/03/vector-similarity-search-vss.html</guid>
<pubDate>Fri, 03 May 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>duckplyr: dplyr Powered by DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/04/02/duckplyr.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/04/02/duckplyr.html</guid>
<pubDate>Tue, 02 Apr 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>No Memory? No Problem. External Aggregation in DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/03/29/external-aggregation.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/03/29/external-aggregation.html</guid>
<pubDate>Fri, 29 Mar 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>42.parquet – A Zip Bomb for the Big Data Age</title>
<description></description>
<link>https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-big-data-age.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/03/26/42-parquet-a-zip-bomb-for-the-big-data-age.html</guid>
<pubDate>Tue, 26 Mar 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Dependency Management in DuckDB Extensions</title>
<description></description>
<link>https://duckdb.org/2024/03/22/dependency-management.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/03/22/dependency-management.html</guid>
<pubDate>Fri, 22 Mar 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>SQL Gymnastics: Bending SQL into Flexible New Shapes</title>
<description></description>
<link>https://duckdb.org/2024/03/01/sql-gymnastics.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/03/01/sql-gymnastics.html</guid>
<pubDate>Fri, 01 Mar 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 0.10.0</title>
<description></description>
<link>https://duckdb.org/2024/02/13/announcing-duckdb-0100.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/02/13/announcing-duckdb-0100.html</guid>
<pubDate>Tue, 13 Feb 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Multi-Database Support in DuckDB</title>
<description></description>
<link>https://duckdb.org/2024/01/26/multi-database-support-in-duckdb.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/01/26/multi-database-support-in-duckdb.html</guid>
<pubDate>Fri, 26 Jan 2024 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Extensions for DuckDB-Wasm</title>
<description></description>
<link>https://duckdb.org/2023/12/18/duckdb-extensions-in-wasm.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/12/18/duckdb-extensions-in-wasm.html</guid>
<pubDate>Mon, 18 Dec 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Updates to the H2O.ai db-benchmark!</title>
<description></description>
<link>https://duckdb.org/2023/11/03/db-benchmark-update.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/11/03/db-benchmark-update.html</guid>
<pubDate>Fri, 03 Nov 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB's CSV Sniffer: Automatic Detection of Types and Dialects</title>
<description></description>
<link>https://duckdb.org/2023/10/27/csv-sniffer.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/10/27/csv-sniffer.html</guid>
<pubDate>Fri, 27 Oct 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckCon #4 in Amsterdam</title>
<description></description>
<link>https://duckdb.org/2023/10/06/duckcon4.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/10/06/duckcon4.html</guid>
<pubDate>Fri, 06 Oct 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 0.9.0</title>
<description></description>
<link>https://duckdb.org/2023/09/26/announcing-duckdb-090.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/09/26/announcing-duckdb-090.html</guid>
<pubDate>Tue, 26 Sep 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB's AsOf Joins: Fuzzy Temporal Lookups</title>
<description></description>
<link>https://duckdb.org/2023/09/15/asof-joins-fuzzy-temporal-lookups.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/09/15/asof-joins-fuzzy-temporal-lookups.html</guid>
<pubDate>Fri, 15 Sep 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Even Friendlier SQL with DuckDB</title>
<description></description>
<link>https://duckdb.org/2023/08/23/even-friendlier-sql.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/08/23/even-friendlier-sql.html</guid>
<pubDate>Wed, 23 Aug 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB ADBC – Zero-Copy Data Transfer via Arrow Database Connectivity</title>
<description></description>
<link>https://duckdb.org/2023/08/04/adbc.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/08/04/adbc.html</guid>
<pubDate>Fri, 04 Aug 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>From Waddle to Flying: Quickly Expanding DuckDB's Functionality with Scalar Python UDFs</title>
<description></description>
<link>https://duckdb.org/2023/07/07/python-udf.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/07/07/python-udf.html</guid>
<pubDate>Fri, 07 Jul 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Correlated Subqueries in SQL</title>
<description></description>
<link>https://duckdb.org/2023/05/26/correlated-subqueries-in-sql.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/05/26/correlated-subqueries-in-sql.html</guid>
<pubDate>Fri, 26 May 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 0.8.0</title>
<description></description>
<link>https://duckdb.org/2023/05/17/announcing-duckdb-080.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/05/17/announcing-duckdb-080.html</guid>
<pubDate>Wed, 17 May 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>10 000 Stars on GitHub</title>
<description></description>
<link>https://duckdb.org/2023/05/12/github-10k-stars.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/05/12/github-10k-stars.html</guid>
<pubDate>Fri, 12 May 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>PostGEESE? Introducing The DuckDB Spatial Extension</title>
<description></description>
<link>https://duckdb.org/2023/04/28/spatial.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/04/28/spatial.html</guid>
<pubDate>Fri, 28 Apr 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckCon #3 in San Francisco</title>
<description></description>
<link>https://duckdb.org/2023/04/28/duckcon3.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/04/28/duckcon3.html</guid>
<pubDate>Fri, 28 Apr 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Introducing DuckDB for Swift</title>
<description></description>
<link>https://duckdb.org/2023/04/21/swift.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/04/21/swift.html</guid>
<pubDate>Fri, 21 Apr 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>The Return of the H2O.ai Database-like Ops Benchmark</title>
<description></description>
<link>https://duckdb.org/2023/04/14/h2oai.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/04/14/h2oai.html</guid>
<pubDate>Fri, 14 Apr 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Shredding Deeply Nested JSON, One Vector at a Time</title>
<description></description>
<link>https://duckdb.org/2023/03/03/json.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/03/03/json.html</guid>
<pubDate>Fri, 03 Mar 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>JupySQL Plotting with DuckDB</title>
<description></description>
<link>https://duckdb.org/2023/02/24/jupysql.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/02/24/jupysql.html</guid>
<pubDate>Fri, 24 Feb 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 0.7.0</title>
<description></description>
<link>https://duckdb.org/2023/02/13/announcing-duckdb-070.html</link>
<guid isPermaLink="false">https://duckdb.org/2023/02/13/announcing-duckdb-070.html</guid>
<pubDate>Mon, 13 Feb 2023 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckCon 2023 – 2nd edition</title>
<description></description>
<link>https://duckdb.org/2022/11/25/duckcon.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/11/25/duckcon.html</guid>
<pubDate>Fri, 25 Nov 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Announcing DuckDB 0.6.0</title>
<description></description>
<link>https://duckdb.org/2022/11/14/announcing-duckdb-060.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/11/14/announcing-duckdb-060.html</guid>
<pubDate>Mon, 14 Nov 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Lightweight Compression in DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/10/28/lightweight-compression.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/10/28/lightweight-compression.html</guid>
<pubDate>Fri, 28 Oct 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Modern Data Stack in a Box with DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/10/12/modern-data-stack-in-a-box.html</guid>
<pubDate>Wed, 12 Oct 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Querying Postgres Tables Directly From DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/09/30/postgres-scanner.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/09/30/postgres-scanner.html</guid>
<pubDate>Fri, 30 Sep 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Persistent Storage of Adaptive Radix Trees in DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/07/27/art-storage.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/07/27/art-storage.html</guid>
<pubDate>Wed, 27 Jul 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Range Joins in DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/05/27/iejoin.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/05/27/iejoin.html</guid>
<pubDate>Fri, 27 May 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Friendlier SQL with DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/05/04/friendlier-sql.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/05/04/friendlier-sql.html</guid>
<pubDate>Wed, 04 May 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Parallel Grouped Aggregation in DuckDB</title>
<description></description>
<link>https://duckdb.org/2022/03/07/aggregate-hashtable.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/03/07/aggregate-hashtable.html</guid>
<pubDate>Mon, 07 Mar 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB Time Zones: Supporting Calendar Extensions</title>
<description></description>
<link>https://duckdb.org/2022/01/06/time-zones.html</link>
<guid isPermaLink="false">https://duckdb.org/2022/01/06/time-zones.html</guid>
<pubDate>Thu, 06 Jan 2022 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB Quacks Arrow: A Zero-copy Data Integration between Apache Arrow and DuckDB</title>
<description></description>
<link>https://duckdb.org/2021/12/03/duck-arrow.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/12/03/duck-arrow.html</guid>
<pubDate>Fri, 03 Dec 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB – The Lord of Enums: The Fellowship of the Categorical and Factors</title>
<description></description>
<link>https://duckdb.org/2021/11/26/duck-enum.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/11/26/duck-enum.html</guid>
<pubDate>Fri, 26 Nov 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Fast Moving Holistic Aggregates</title>
<description></description>
<link>https://duckdb.org/2021/11/12/moving-holistic.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/11/12/moving-holistic.html</guid>
<pubDate>Fri, 12 Nov 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>DuckDB-Wasm: Efficient Analytical SQL in the Browser</title>
<description></description>
<link>https://duckdb.org/2021/10/29/duckdb-wasm.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/10/29/duckdb-wasm.html</guid>
<pubDate>Fri, 29 Oct 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Windowing in DuckDB</title>
<description></description>
<link>https://duckdb.org/2021/10/13/windowing.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/10/13/windowing.html</guid>
<pubDate>Wed, 13 Oct 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Fastest Table Sort in the West – Redesigning DuckDB’s Sort</title>
<description></description>
<link>https://duckdb.org/2021/08/27/external-sorting.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/08/27/external-sorting.html</guid>
<pubDate>Fri, 27 Aug 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Querying Parquet with Precision Using DuckDB</title>
<description></description>
<link>https://duckdb.org/2021/06/25/querying-parquet.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/06/25/querying-parquet.html</guid>
<pubDate>Fri, 25 Jun 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Efficient SQL on Pandas with DuckDB</title>
<description></description>
<link>https://duckdb.org/2021/05/14/sql-on-pandas.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/05/14/sql-on-pandas.html</guid>
<pubDate>Fri, 14 May 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
<item>
<title>Testing Out DuckDB's Full Text Search Extension</title>
<description></description>
<link>https://duckdb.org/2021/01/25/full-text-search.html</link>
<guid isPermaLink="false">https://duckdb.org/2021/01/25/full-text-search.html</guid>
<pubDate>Mon, 25 Jan 2021 00:00:00 GMT</pubDate>
<author>DuckDB Organization</author>
</item>
</channel>
</rss> |
TonyRL
reviewed
Jul 16, 2024
|
Successfully generated as following: http://localhost:1200/duckdb/news - Success ✔️<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
<channel>
<title>DuckDB News</title>
<link>https://duckdb.org/news/</link>
<atom:link href="http://localhost:1200/duckdb/news" rel="self" type="application/rss+xml"></atom:link>
<description>DuckDB News - Made with love by RSSHub(https://github.com/DIYgod/RSSHub)</description>
<generator>RSSHub</generator>
<webMaster>i@diygod.me (DIYgod)</webMaster>
<language>en</language>
<lastBuildDate>Tue, 16 Jul 2024 13:16:26 GMT</lastBuildDate>
<ttl>5</ttl>
<item>
<title>DuckCon #5 in Seattle</title>
<description><div class="eventdate">2024-08-15</div>
<h1>DuckCon #5 in Seattle</h1>
<div class="infoline">
<div class="icon">
</div>
<div><span class="author">Mark Raasveldt, Hannes Mühleisen, Gabor Szarnyas, Kelly de Smit</span></div>
</div>
<p><img src="https://duckdb.org/images/duckcon5-splashscreen.svg" alt="DuckCon #5 Splashscreen" width="680" referrerpolicy="no-referrer"></p>
<p>We are excited to hold the next "DuckCon" DuckDB user group meeting in <strong>Seattle, WA</strong>, sponsored by <a href="https://motherduck.com/">MotherDuck</a>.
The meeting will take place on August 15, 2024 (Thursday) in the <a href="https://www.siff.net/cinema/cinema-venues/siff-cinema-egyptian">SIFF Cinema Egyptian</a>, located at <a href="https://maps.app.goo.gl/jRfRPMaYY6AmJ2fF6">805 E Pine St, Seattle, WA, 98122</a>.</p>
<p>As is traditional in DuckCons, we will start with a talk from DuckDB's creators <a href="https://hannes.muehleisen.org/">Hannes Mühleisen</a> and <a href="https://mytherin.github.io/">Mark Raasveldt</a> about the state of DuckDB. This will be followed by presentations by DuckDB users. In addition, we will have several lightning talks from the DuckDB community.</p>
<h3 id="timetable">
<a style="text-decoration: none;" href="https://duckdb.org/2024/08/15/duckcon5.html#timetable">Timetable</a>
</h3>
<!-- To watch the recordings, see the [playlist of talks](https://www.youtube.com/playlist?list=). -->
<table>
<thead>
<tr>
<th>Time</th>
<th style="text-align: left">Title</th>
<th style="text-align: left">Presenter</th>
</tr>
</thead>
<tbody>
<tr>
<td>1:30PM</td>
<td style="text-align: left"><em>First session</em></td>
<td style="text-align: left">&nbsp;</td>
</tr>
<tr>
<td>1:30PM</td>
<td style="text-align: left"><strong>Introductions</strong></td>
<td style="text-align: left"><a href="https://hannes.muehleisen.org/">Hannes Mühleisen</a> <br> <em>(<a href="https://duckdblabs.com/">DuckDB Labs</a>)</em></td>
</tr>
<tr>
<td>1:40PM</td>
<td style="text-align: left"><strong>Overview and latest developments</strong></td>
<td style="text-align: left">Hannes Mühleisen, <a href="https://mytherin.github.io/">Mark Raasveldt</a> <br> <em>(DuckDB Labs)</em></td>
</tr>
<tr>
<td>2:10PM</td>
<td style="text-align: left"><strong>MotherDuck: Taking flight with interactive analytics</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/frances-perry/">Frances Perry</a> <br> <em>(<a href="https://motherduck.com/">MotherDuck</a>)</em></td>
</tr>
<tr>
<td>2:40PM</td>
<td style="text-align: left"><strong>How DuckDB is changing the face of spatial analytics</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/mbforr/">Matt Forrest</a> <br> <em>(<a href="https://carto.com/">CARTO</a>)</em></td>
</tr>
<tr>
<td>3:10PM</td>
<td style="text-align: left"><em>Break</em></td>
<td style="text-align: left">&nbsp;</td>
</tr>
<tr>
<td>3:30PM</td>
<td style="text-align: left"><em>Second session</em></td>
<td style="text-align: left">&nbsp;</td>
</tr>
<tr>
<td>3:30PM</td>
<td style="text-align: left"><strong>A duck for your dashboard: Performant data apps in the browser with DuckDB</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/rkosara/">Robert Kosara</a> <br> <em>(<a href="https://observablehq.com/">Observable</a>)</em></td>
</tr>
<tr>
<td>4:00PM</td>
<td style="text-align: left">Lightning talks</td>
<td style="text-align: left">&nbsp;</td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>Delighting users with RESTful APIs and DuckDB</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/miguelmfilipe/">Miguel Filipe</a> <br> <em>(<a href="https://dune.com/">Dune Analytics</a>)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>Aerodynamic data models: Flying fast at scale with DuckDB</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/begelundmuller/">Benjamin Egelund-Müller</a> <br> <em>(<a href="https://www.rilldata.com/">Rill Data</a>)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>Scaling business intelligence in the built environment using analytics and AI to improve health</strong></td>
<td style="text-align: left"><a href="https://jaan.io/">Jaan Lı</a> <br> <em>(<a href="https://ut.ee/en/home">University of Tartu</a> and <a href="https://www.onefact.org/">One Fact Foundation</a>)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>Double glazing: Two years of windowing improvements</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/riwesley/">Richard Wesley</a> <br> <em>(DuckDB Labs)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>dbverse: composable database libraries for larger-than-memory scientific analytics</strong></td>
<td style="text-align: left"><a href="https://twitter.com/Ed2uiz">Edward Ruiz</a> <br> <em>(<a href="https://www.bu.edu/">Boston University</a>)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>A quack at building scalable data pipelines with DuckDB</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/junaidrahim/">Junaid Rahim</a> <br> <em>(<a href="https://atlan.com/">Atlan</a>)</em></td>
</tr>
<tr>
<td>&nbsp;</td>
<td style="text-align: left"><strong>Quack attack: Bringing DuckDB to the dart side</strong></td>
<td style="text-align: left"><a href="https://www.linkedin.com/in/andyprock/">Andy Prock</a> <br> <em>(<a href="https://www.tigereye.com/">TigerEye</a>)</em></td>
</tr>
<tr>
<td>5:00PM</td>
<td style="text-align: left">Drinks and snacks</td>
<td style="text-align: left">&nbsp;</td>
</tr>
<tr>
<td>6:30PM</td>
<td style="text-align: left"><em>End of event</em></td>
<td style="text-align: left">&nbsp;</td>
</tr>
</tbody>
</table>
<h3 id="availability-of-talks">
<a style="text-decoration: none;" href="https://duckdb.org/2024/08/15/duckcon5.html#availability-of-talks">Availability of Talks</a>
</h3>
<p>The event will not be streamed online and there is no hybrid option for participation.
However, the slides and recorded videos of the talks will be published a few weeks after the event, similarly to <a href="https://duckdb.org/2023/04/28/duckcon3.html">DuckCon #3</a> and <a href="https://duckdb.org/2023/10/06/duckcon4.html">DuckCon #4</a>.</p>
<h3 id="registration-process">
<a style="text-decoration: none;" href="https://duckdb.org/2024/08/15/duckcon5.html#registration-process">Registration Process</a>
</h3>
<p>Attendance is free. While supplies last, you can still get a ticket on <a href="https://www.eventbrite.com/e/duckcon-5-tickets-877957674037">Eventbrite</a>.
You will need to show this ticket at the entrance to attend.</p>
<p><strong>If you register before July 24, you will get a badge with your name at the registration desk.</strong></p>
<h3 id="parking-information">
<a style="text-decoration: none;" href="https://duckdb.org/2024/08/15/duckcon5.html#parking-information">Parking Information</a>
</h3>
<ul>
<li>Parking garage is available across the street at <a href="https://maps.app.goo.gl/dWe76SbhGtZ2j9Dz7">1609 Harvard Ave</a> for $15/day.</li>
<li>There is no cloakroom and there are no lockers available.</li>
</ul>
<h3 id="inquiries">
<a style="text-decoration: none;" href="https://duckdb.org/2024/08/15/duckcon5.html#inquiries">Inquiries</a>
</h3>
<p>Please contact Kelly de Smit at <a href="https://duckdb.org/cdn-cgi/l/email-protection#7d16181111043d19081e16191f111c1f0e531e1210"><span class="__cf_email__" data-cfemail="701b151c1c09301405131b14121c1112035e131f1d">[email&nbsp;protected]</span></a> if you have any questions.</p>
</description>
<link>https://duckdb.org/2024/08/15/duckcon5.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/08/15/duckcon5.html</guid>
<pubDate>Thu, 15 Aug 2024 00:00:00 GMT</pubDate>
<author>Mark Raasveldt, Hannes Mühleisen, Gabor Szarnyas, Kelly de Smit</author>
</item>
<item>
<title>Memory Management in DuckDB</title>
<description><h1>Memory Management in DuckDB</h1>
<div class="infoline">
<div class="icon">
<img src="https://duckdb.org/images/blog/authors/mark_raasveldt.jpg" alt="Author Avatar" referrerpolicy="no-referrer">
</div>
<div><span class="author">Mark Raasveldt</span><span>2024-07-09</span></div>
</div>
<p>Memory is an important resource when processing large amounts of data. Memory is a fast caching layer that can provide immense speed-ups to query processing. However, memory is finite and expensive, and when working with large data sets there is generally not enough memory available to keep all necessary data structures cached. Managing memory effectively is critical for a high-performance query engine – as memory must be utilized in order to provide that high performance, but we must be careful so that we do not use excessive memory which can cause out-of-memory errors or can cause the ominous <a href="https://en.wikipedia.org/wiki/Out_of_memory#Recovery">OOM killer</a> to zap the process out of existence.</p>
<p>DuckDB is built to effectively utilize available memory while avoiding running out of memory:</p>
<ul>
<li>The streaming execution engine allows small chunks of data to flow through the system without requiring entire data sets to be materialized in memory.</li>
<li>Data from intermediates can be spilled to disk temporarily in order to free up space in memory, allowing computation of complex queries that would otherwise exceed the available memory.</li>
<li>The buffer manager caches as many pages as possible from any attached databases without exceeding the pre-defined memory limits.</li>
</ul>
<p>In this blog post we will cover these aspects of memory management within DuckDB – and provide examples of where they are utilized.</p>
<h2 id="streaming-execution">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/09/memory-management.html#streaming-execution">Streaming Execution</a>
</h2>
<p>DuckDB uses a streaming execution engine to process queries. Data sources, such as tables, CSV files or Parquet files, are never fully materialized in memory. Instead, data is read and processed one chunk at a time. For example, consider the execution of the following query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">UserAgent</span><span class="p">,</span>
<span class="nf">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
<span class="k">FROM</span> <span class="s1">'hits.csv'</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">UserAgent</span><span class="p">;</span>
</code></pre></div></div>
<p>Instead of reading the entire CSV file at once, DuckDB reads data from the CSV file in pieces, and computes the aggregation incrementally using the data read from those pieces. This happens continuously until the entire CSV file is read, at which point the entire aggregation result is computed.</p>
<p><img src="https://duckdb.org/images/blog/streamingexecution.png" alt="DuckDB Streaming Execution" width="800" referrerpolicy="no-referrer"></p>
<p>In the above example we are only showing a single data stream. In practice, DuckDB uses multiple data streams to enable multi-threaded execution – each thread executes its own data stream. The aggregation results of the different threads are combined to compute the final result.</p>
<p>While streaming execution is conceptually simple, it is powerful, and is sufficient to provide larger-than-memory support for many simple use cases. For example, streaming execution enables larger-than-memory support for:</p>
<ul>
<li>Computing aggregations where the total number of groups is small</li>
<li>Reading data from one file and writing to another (e.g., reading from CSV and writing to Parquet)</li>
<li>Computing a Top-N over the data (where N is small)</li>
</ul>
<p>Note that nothing needs to be done to enable streaming execution – DuckDB always processes queries in this manner.</p>
<h2 id="intermediate-spilling">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/09/memory-management.html#intermediate-spilling">Intermediate Spilling</a>
</h2>
<p>While streaming execution enables larger-than-memory processing for simple queries, there are many cases where streaming execution alone is not sufficient.</p>
<p>In the previous example, streaming execution enabled larger-than-memory processing because the computed aggregate result was very small – as there are very few unique user agents in comparison to the total number of web requests. As a result, the aggregate hash table would always remain small, and never exceed the amount of available memory.</p>
<p>Streaming execution is not sufficient if the intermediates required to process a query are larger than memory. For example, suppose we group by the source IP in the previous example:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">IPNetworkID</span><span class="p">,</span>
<span class="nf">count</span><span class="p">(</span><span class="o">*</span><span class="p">)</span>
<span class="k">FROM</span> <span class="s1">'hits.csv'</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">IPNetworkID</span><span class="p">;</span>
</code></pre></div></div>
<p>Since there are many more unique source IPs, the hash table we need to maintain is significantly larger. If the size of the aggregate hash table exceeds memory, the streaming execution engine is not sufficient to prevent out-of-memory issues.</p>
<p>Larger-than-memory intermediates can happen in many scenarios, in particular when executing more complex queries. For example, the following scenarios can lead to larger-than-memory intermediates:</p>
<ul>
<li>Computing an aggregation with many unique groups</li>
<li>Computing an exact distinct count of a column with many distinct values</li>
<li>Joining two tables together that are both larger than memory</li>
<li>Sorting a larger-than-memory dataset</li>
<li>Computing a complex window over a larger-than-memory table</li>
</ul>
<p>DuckDB deals with these scenarios by disk spilling. Larger-than-memory intermediates are (partially) written to disk in the temporary directory when required. While powerful, disk spilling reduces performance – as additional I/O must be performed. For that reason, DuckDB tries to minimize disk spilling. Disk spilling is adaptively used only when the size of the intermediates increases past the memory limit. Even in those scenarios, as much data is kept in memory as possible to maximize performance. The exact way this is done depends on the operators and is detailed in other blog posts
(<a href="https://duckdb.org/2024/03/29/external-aggregation.html">aggregation</a>,
<a href="https://duckdb.org/2021/08/27/external-sorting.html">sorting</a>).</p>
<p>The <code class="language-plaintext highlighter-rouge">memory_limit</code> setting controls how much data DuckDB is allowed to keep in memory. By default, this is set to <code class="language-plaintext highlighter-rouge">80%</code> of the physical RAM of your system (e.g., if your system has 16 GB RAM, this defaults to 12.8 GB). The memory limit can be changed using the following command:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SET</span> <span class="py">memory_limit</span> <span class="o">=</span> <span class="s1">'4GB'</span><span class="p">;</span>
</code></pre></div></div>
<p>The location of the temporary directory can be chosen using the <code class="language-plaintext highlighter-rouge">temp_directory</code> setting, and is by default the connected database with a <code class="language-plaintext highlighter-rouge">.tmp</code> suffix (e.g., <code class="language-plaintext highlighter-rouge">database.db.tmp</code>), or only <code class="language-plaintext highlighter-rouge">.tmp</code> if connecting to an in-memory database. The maximum size of the temporary directory can be limited using the <code class="language-plaintext highlighter-rouge">max_temp_directory_size</code> setting, which defaults to <code class="language-plaintext highlighter-rouge">90%</code> of the remaining disk space on the drive where the temporary files are stored. These settings can be adjusted as follows:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SET</span> <span class="py">temp_directory</span> <span class="o">=</span> <span class="s1">'/tmp/duckdb_swap'</span><span class="p">;</span>
<span class="k">SET</span> <span class="py">max_temp_directory_size</span> <span class="o">=</span> <span class="s1">'100GB'</span><span class="p">;</span>
</code></pre></div></div>
<p>If the memory limit is exceeded and disk spilling cannot be used, either because disk spilling is explicitly disabled, the temporary directory size exceeds the provided limit, or a system limitation means that disk spilling cannot be used for a given query – an out-of-memory error is reported and the query is canceled.</p>
<h2 id="buffer-manager">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/09/memory-management.html#buffer-manager">Buffer Manager</a>
</h2>
<p>Another core component of memory management in DuckDB is the buffer manager. The buffer manager is responsible for caching pages from DuckDB's own persistent storage. Conceptually the buffer manager works in a similar fashion to the intermediate spilling. Pages are kept in memory as much as possible, and evicted from memory when space is required for other data structures. The buffer manager abides by the same memory limit as any intermediate data structures. Pages in the buffer manager can be freed up to make space for intermediate data structures, or vice versa.</p>
<p>There are two main differences between the buffer manager and intermediate data structures:</p>
<ul>
<li>As the buffer manager caches pages that already exist on disk (in DuckDB's persistent storage) – they do not need to be written to the temporary directory when evicted. Instead, when they are required again, they can be re-read from the attached storage file directly.</li>
<li>Query intermediates have a natural life-cycle, namely when the query is finished processing the intermediates are no longer required. Pages that are buffer managed from the persistent storage are useful across queries. As such, the pages kept by the buffer manager are kept cached until either the persistent database is closed, or until space must be freed up for other operations.</li>
</ul>
<p>The performance boost of the buffer manager depends on the speed of the underlying storage medium. When data is stored on a very fast disk, reading data is fast and the speed-up is minimal. When data is stored on a network drive or read over http/S3, reading requires performing network requests, and the speed-up can be very large.</p>
<h2 id="profiling-memory-usage">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/09/memory-management.html#profiling-memory-usage">Profiling Memory Usage</a>
</h2>
<p>DuckDB contains a number of tools that can be used to profile memory usage.</p>
<p>The <code class="language-plaintext highlighter-rouge">duckdb_memory()</code> function can be used to inspect which components of the system are using memory. Memory used by the buffer manager is labeled as <code class="language-plaintext highlighter-rouge">BASE_TABLE</code>, while query intermediates are divided into separate groups.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span> <span class="nf">duckdb_memory</span><span class="p">();</span>
</code></pre></div></div>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>┌──────────────────┬────────────────────┬─────────────────────────┐
│ tag │ memory_usage_bytes │ temporary_storage_bytes │
│ varchar │ int64 │ int64 │
├──────────────────┼────────────────────┼─────────────────────────┤
│ BASE_TABLE │ 168558592 │ 0 │
│ HASH_TABLE │ 0 │ 0 │
│ PARQUET_READER │ 0 │ 0 │
│ CSV_READER │ 0 │ 0 │
│ ORDER_BY │ 0 │ 0 │
│ ART_INDEX │ 0 │ 0 │
│ COLUMN_DATA │ 0 │ 0 │
│ METADATA │ 0 │ 0 │
│ OVERFLOW_STRINGS │ 0 │ 0 │
│ IN_MEMORY_TABLE │ 0 │ 0 │
│ ALLOCATOR │ 0 │ 0 │
│ EXTENSION │ 0 │ 0 │
├──────────────────┴────────────────────┴─────────────────────────┤
│ 12 rows 3 columns │
└─────────────────────────────────────────────────────────────────┘
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">duckdb_temporary_files</code> function can be used to examine the current contents of the temporary directory.</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">FROM</span> <span class="nf">duckdb_temporary_files</span><span class="p">();</span>
</code></pre></div></div>
<div class="language-text highlighter-rouge"><div class="highlight"><pre class="highlight"><code>┌────────────────────────────────┬───────────┐
│ path │ size │
│ varchar │ int64 │
├────────────────────────────────┼───────────┤
│ .tmp/duckdb_temp_storage-0.tmp │ 967049216 │
└────────────────────────────────┴───────────┘
</code></pre></div></div>
<h2 id="conclusion">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/09/memory-management.html#conclusion">Conclusion</a>
</h2>
<p>Memory management is critical for a high-performance analytics engine. DuckDB is built to take advantage of any available memory to speed up query processing, while gracefully dealing with larger-than-memory datasets using intermediate spilling. Memory management is still an active area of development and has <a href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#scale-tests">continuously improved across DuckDB versions</a>. Amongst others, we are working on improving memory management for complex queries that involve multiple operators with larger-than-memory intermediates.</p>
</description>
<link>https://duckdb.org/2024/07/09/memory-management.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/07/09/memory-management.html</guid>
<pubDate>Tue, 09 Jul 2024 00:00:00 GMT</pubDate>
<author>Mark Raasveldt</author>
</item>
<item>
<title>DuckDB Community Extensions</title>
<description><h1>DuckDB Community Extensions</h1>
<div class="infoline">
<div class="icon">
</div>
<div><span class="author">The DuckDB team</span><span>2024-07-05</span></div>
</div>
<div class="excerpt"><p><em>TL;DR: DuckDB extensions can now be published via the <a href="https://github.com/duckdb/community-extensions">DuckDB Community Extensions repository</a>. The repository makes it easier for users to install extensions using the <code class="language-plaintext highlighter-rouge">INSTALL ⟨extension name⟩ FROM community</code> syntax. Extension developers avoid the burdens of compilation and distribution.</em></p>
</div>
<h2 id="duckdb-extensions">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#duckdb-extensions">DuckDB Extensions</a>
</h2>
<h3 id="design-philosophy">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#design-philosophy">Design Philosophy</a>
</h3>
<p>One of the main design goals of DuckDB is <em>simplicity</em>, which – to us – implies that the system should be rather nimble, very light on dependencies, and generally small enough to run on constrained platforms like <a href="https://duckdb.org/docs/api/wasm/overview.html">WebAssembly</a>. This goal is in direct conflict with very reasonable user requests to support advanced features like spatial data analysis, vector indexes, connectivity to various other databases, support for data formats, etc. Baking all those features into a monolithic binary is certainly possible and the route some systems take. But we want to preserve DuckDB’s simplicity. Also, shipping all possible features would be quite excessive for most users because no use cases require <em>all</em> extensions at the same time (the “Microsoft Word paradox”, where even power users only use a few features of the system, but the exact set of features vary between users).</p>
<p>To achieve this, DuckDB has a powerful extension mechanism, which allows users to add new functionalities to DuckDB. This mechanism allows for registering new functions, supporting new file formats and compression methods, handling new network protocols, etc. In fact, many of DuckDB’s popular features are implemented as extensions: the <a href="https://duckdb.org/docs/data/parquet/overview.html">Parquet reader</a>, the <a href="https://duckdb.org/docs/extensions/json.html">JSON reader</a>, and the <a href="https://duckdb.org/docs/extensions/httpfs/overview.html">HTTPS/S3 connector</a> all use the extension mechanism.</p>
<h3 id="using-extensions">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#using-extensions">Using Extensions</a>
</h3>
<p>Since <a href="https://github.com/duckdb/duckdb/releases/tag/v0.3.2">version 0.3.2</a>, we have already greatly simplified the discovery and installation by hosting them on a centralized extension repository. So, for example, to install the <a href="https://duckdb.org/docs/extensions/spatial.html">spatial extension</a>, one can just run the following commands using DuckDB’s SQL interface:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSTALL</span> <span class="n">spatial</span><span class="p">;</span> <span class="c1">-- once</span>
<span class="k">LOAD</span> <span class="n">spatial</span><span class="p">;</span> <span class="c1">-- on each use</span>
</code></pre></div></div>
<p>What happens behind the scenes is that DuckDB downloads an extension binary suitable to the current operating system and processor architecture (e.g., macOS on ARM64) and stores it in the <code class="language-plaintext highlighter-rouge">~/.duckdb</code> folder. On each <code class="language-plaintext highlighter-rouge">LOAD</code>, this file is loaded into the running DuckDB instance, and things happily continue from there. Of course, for this to work, we compile, sign and host the extensions for a rather large and growing list of processor architecture – operating system combinations. This mechanism is already heavily used, currently, we see around six million extension downloads <em>each week</em> with a corresponding data transfer volume of around 40 terabytes!</p>
<p>Until now, publishing third-party extensions has been a <em>difficult process</em> which required the extension developer to build the extensions in their repositories for a host of platforms. Moreover, they were unable to sign the extensions using official keys, forcing users to use the <code class="language-plaintext highlighter-rouge">allow_unsigned_extensions</code> option that disables signature checks which is problematic in itself.</p>
<h2 id="duckdb-community-extensions">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#duckdb-community-extensions">DuckDB Community Extensions</a>
</h2>
<p>Distributing software in a safe way has never been easier, allowing us to reach a wide base of users across pip, conda, cran, npm, brew, etc. We want to provide a similar experience both to users who can easily grab the extension they will want to use, and developers who should not be burdened with distribution details. We are also interested in lowering the bar to package utilities and scripts as a DuckDB extension, empowering users to package useful functionality connected to their area of expertise (or pain points).</p>
<p>We believe that fostering a community extension ecosystem is the next logical step for DuckDB. That’s why we’re very excited about launching our <a href="https://github.com/duckdb/community-extensions/">Community Extension repository</a> which was <a href="https://youtu.be/wuP6iEYH11E?t=275">announced at the Data + AI Summit</a>.</p>
<p>For users, this repository allows for easy discovery, installation and maintenance of community extensions directly from the DuckDB SQL prompt. For developers, it greatly streamlines the publication process of extensions. In the following, we’ll discuss how the new extension repository enhances the experiences of these groups.</p>
<h3 id="user-experience">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#user-experience">User Experience</a>
</h3>
<p>We are going to use the <a href="https://github.com/isaacbrodsky/h3-duckdb"><code class="language-plaintext highlighter-rouge">h3</code> extension</a> as our example. This extension implements <a href="https://github.com/uber/h3">hierarchical hexagonal indexing</a> for geospatial data.</p>
<p>Using the DuckDB Community Extensions repository, you can now install and load the <code class="language-plaintext highlighter-rouge">h3</code> extension as follows:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">INSTALL</span> <span class="n">h3</span> <span class="k">FROM</span> <span class="n">community</span><span class="p">;</span>
<span class="k">LOAD</span> <span class="n">h3</span><span class="p">;</span>
</code></pre></div></div>
<p>Then, you can instantly start using it. Note that the sample data is 500 MB:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span>
<span class="nf">h3_latlng_to_cell</span><span class="p">(</span><span class="n">pickup_latitude</span><span class="p">,</span> <span class="n">pickup_longitude</span><span class="p">,</span> <span class="mi">9</span><span class="p">)</span> <span class="k">AS</span> <span class="n">cell_id</span><span class="p">,</span>
<span class="nf">h3_cell_to_boundary_wkt</span><span class="p">(</span><span class="n">cell_id</span><span class="p">)</span> <span class="k">AS</span> <span class="n">boundary</span><span class="p">,</span>
<span class="nf">count</span><span class="p">()</span> <span class="k">AS</span> <span class="n">cnt</span>
<span class="k">FROM</span> <span class="nf">read_parquet</span><span class="p">(</span><span class="s1">'https://blobs.duckdb.org/data/yellow_tripdata_2010-01.parquet'</span><span class="p">)</span>
<span class="k">GROUP</span> <span class="k">BY</span> <span class="n">cell_id</span>
<span class="k">HAVING</span> <span class="n">cnt</span> <span class="o">&gt;</span> <span class="mi">10</span><span class="p">;</span>
</code></pre></div></div>
<p>On load, the extension’s signature is checked, both to ensure platform and versions are compatible, and to verify that the source of the binary is the community extensions repository. Extensions are built, signed and distributed for Linux, macOS, Windows, and WebAssembly. This allows extensions to be available to any DuckDB client using version 1.0.0 and upcoming versions.</p>
<p>The <code class="language-plaintext highlighter-rouge">h3</code> extension’s documentation is available at <a href="https://community-extensions.duckdb.org/extensions/h3.html">https://community-extensions.duckdb.org/extensions/h3.html</a>.</p>
<h3 id="developer-experience">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#developer-experience">Developer Experience</a>
</h3>
<p>From the developer’s perspective, the Community Extensions repository performs the steps required for publishing extensions, including building the extensions for all relevant <a href="https://duckdb.org/docs/dev/building/supported_platforms.html">platforms</a>, signing the extension binaries and serving them from the repository.</p>
<p>For the <a href="https://github.com/isaacbrodsky/">maintainer of <code class="language-plaintext highlighter-rouge">h3</code></a>, the publication process required performing the following steps:</p>
<ol>
<li>
<p>Sending a PR with a metadata file <code class="language-plaintext highlighter-rouge">description.yml</code> contains the description of the extension:</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">extension</span><span class="pi">:</span>
<span class="na">name</span><span class="pi">:</span> <span class="s">h3</span>
<span class="na">description</span><span class="pi">:</span> <span class="s">Hierarchical hexagonal indexing for geospatial data</span>
<span class="na">version</span><span class="pi">:</span> <span class="s">1.0.0</span>
<span class="na">language</span><span class="pi">:</span> <span class="s">C++</span>
<span class="na">build</span><span class="pi">:</span> <span class="s">cmake</span>
<span class="na">license</span><span class="pi">:</span> <span class="s">Apache-2.0</span>
<span class="na">maintainers</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">isaacbrodsky</span>
<span class="na">repo</span><span class="pi">:</span>
<span class="na">github</span><span class="pi">:</span> <span class="s">isaacbrodsky/h3-duckdb</span>
<span class="na">ref</span><span class="pi">:</span> <span class="s">3c8a5358e42ab8d11e0253c70f7cc7d37781b2ef</span>
</code></pre></div> </div>
</li>
<li>
<p>The CI will build and test the extension. The checks performed by the CI are aligned with the <a href="https://github.com/duckdb/extension-template"><code class="language-plaintext highlighter-rouge">extension-template</code> repository</a>, so iterations can be done independently.</p>
</li>
<li>
<p>Wait for approval from the DuckDB Community Extension repository’s maintainers and for the build process to complete.</p>
</li>
</ol>
<h2 id="published-extensions">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#published-extensions">Published Extensions</a>
</h2>
<p>To show that it’s feasible to publish extensions, we reached out to a few developers of key extensions. At the time of the publication of this blog post, the DuckDB Community Extensions repository already contains the following extensions.</p>
<div class="narrow_table"></div>
<table>
<thead>
<tr>
<th>Name</th>
<th>Description</th>
</tr>
</thead>
<tbody>
<tr>
<td><a href="https://github.com/rustyconover/duckdb-crypto-extension">crypto</a></td>
<td>Adds cryptographic hash functions and <a href="https://en.wikipedia.org/wiki/HMAC">HMAC</a>.</td>
</tr>
<tr>
<td><a href="https://github.com/isaacbrodsky/h3-duckdb">h3</a></td>
<td>Implements hierarchical hexagonal indexing for geospatial data.</td>
</tr>
<tr>
<td><a href="https://github.com/rustyconover/duckdb-lindel-extension">lindel</a></td>
<td>Implements linearization/delinearization, Z-Order, Hilbert and Morton curves.</td>
</tr>
<tr>
<td><a href="https://github.com/ywelsch/duckdb-prql">prql</a></td>
<td>Allows running <a href="https://prql-lang.org/">PRQL</a> commands directly within DuckDB.</td>
</tr>
<tr>
<td><a href="https://github.com/pdet/Scrooge-McDuck">scrooge</a></td>
<td>Supports a set of aggregation functions and data scanners for financial data.</td>
</tr>
<tr>
<td><a href="https://github.com/rustyconover/duckdb-shellfs-extension">shellfs</a></td>
<td>Allows shell commands to be used for input and output.</td>
</tr>
</tbody>
</table>
<p>DuckDB Labs and the DuckDB Foundation do not vet the code within community extensions and, therefore, cannot guarantee that DuckDB community extensions are safe to use. The loading of community extensions can be explicitly disabled with the following one-way configuration option:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SET</span> <span class="py">allow_community_extensions</span> <span class="o">=</span> <span class="k">false</span><span class="p">;</span>
</code></pre></div></div>
<p>For more details, see the documentation’s <a href="https://duckdb.org/docs/operations_manual/securing_duckdb/securing_extensions.html#community-extension">Securing DuckDB page</a>.</p>
<h2 id="summary-and-looking-ahead">
<a style="text-decoration: none;" href="https://duckdb.org/2024/07/05/community-extensions.html#summary-and-looking-ahead">Summary and Looking Ahead</a>
</h2>
<p>In this blog post, we introduced the DuckDB Community Extensions repository, which allows easy installation of third-party DuckDB extensions.</p>
<p>We are looking forward to continuously extending this repository. If you have an idea for creating an extension, take a look at the already published extension source codes, which provide good examples of how to package community extensions, and join the <code class="language-plaintext highlighter-rouge">#extensions</code> channel on our <a href="https://discord.duckdb.org/">Discord</a>.
Once you have an extension, please contribute it via a <a href="https://github.com/duckdb/community-extensions/pulls">pull request</a>.</p>
<p>Finally, we would like to thank the early adopters of DuckDB’s extension mechanism and Community Extension repository. Thanks for iterating with us and providing feedback to us.</p>
</description>
<link>https://duckdb.org/2024/07/05/community-extensions.html</link>
<guid isPermaLink="false">https://duckdb.org/2024/07/05/community-extensions.html</guid>
<pubDate>Fri, 05 Jul 2024 00:00:00 GMT</pubDate>
<author>The DuckDB team</author>
</item>
<item>
<title>Benchmarking Ourselves over Time at DuckDB</title>
<description><h1>Benchmarking Ourselves over Time at DuckDB</h1>
<div class="infoline">
<div class="icon">
<img src="https://duckdb.org/images/blog/authors/alex_monahan.jpg" alt="Author Avatar" referrerpolicy="no-referrer">
</div>
<div><span class="author">Alex Monahan</span><span>2024-06-26</span></div>
</div>
<div class="excerpt"><p><em>TL;DR: In the last 3 years, DuckDB has become 3-25x faster and can analyze ~10x larger datasets all on the same hardware.</em></p>
</div>
<!-- <script src="https://cdn.plot.ly/plotly-latest.min.js"></script> -->
<div id="overall_results_by_time_header" style="width:100%;height:400px;"></div>
<p>A big part of DuckDB's focus is on the developer experience of working with data.
However, performance is an important consideration when investigating data management systems.
Fairly comparing data processing systems using benchmarks is <a href="https://mytherin.github.io/papers/2018-dbtest.pdf">very difficult</a>.
Whoever creates the benchmark is likely to know one system better than the rest, influencing benchmark selection, how much time is spent tuning parameters, and more.</p>
<p>Instead, this post focuses on benchmarking <em>our own</em> performance over time.
<!-- (Of course, we encourage you to conduct your own benchmarks and welcome your feedback on our [Discord server](https://discord.duckdb.org/) or in [GitHub discussions](https://github.com/duckdb/duckdb/discussions)!). -->
This approach avoids many comparison pitfalls, and also provides several valuable data points to consider when selecting a system.</p>
<ul>
<li>
<p><strong>How fast is it improving?</strong>
Learning a new tool is an investment.
Picking a vibrant, rapidly improving database ensures your choice pays dividends for years to come.
Plus, if you haven't experimented with a tool in a while, you can see how much faster it has become since you last checked!</p>
</li>
<li>
<p><strong>What is it especially good at?</strong>
The choice of benchmark is an indicator of what types of workloads a tool is useful for.
The higher the variety of analyses in the benchmark, the more broadly useful the tool can be.</p>
</li>
<li>
<p><strong>What scale of data can it handle?</strong>
Many benchmarks are deliberately smaller than typical workloads.
This allows the benchmark to complete in a reasonable amount of time when run with many configurations.
However, an important question to answer when selecting a system is whether the size of your data can be handled within the size of your compute resources.</p>
</li>
</ul>
<!-- #### Limitations of Benchmarking over Time -->
<p>There are some limitations when looking at the performance of a system over time.
If a feature is brand new, there is no prior performance to compare to!
As a result, this post focuses on fundamental workloads rather than DuckDB's ever-increasing set of integrations with different lakehouse data formats, cloud services, and more.</p>
<p>The code used to run the benchmark also avoids many of DuckDB's <a href="https://duckdb.org/docs/sql/dialect/friendly_sql.html">Friendlier SQL</a> additions, as those have also been added more recently.
(When writing these queries, it felt like going back in time!)</p>
<h2 id="benchmark-design-summary">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#benchmark-design-summary">Benchmark Design Summary</a>
</h2>
<p>This post measures DuckDB's performance over time using the <a href="https://duckdblabs.github.io/db-benchmark/">H2O.ai benchmark</a>, plus some new benchmarks added for importing, exporting, and using window functions.
Please see our previous <a href="https://duckdb.org/2023/04/14/h2oai.html">blog</a> <a href="https://duckdb.org/2023/11/03/db-benchmark-update.html">posts</a> for details on why we believe the H2O.ai benchmark is a good approach! The full details of the benchmark design are in the appendix.</p>
<ul>
<li>H2O.ai, plus import/export and window function tests</li>
<li>Python instead of R</li>
<li>5GB scale for everything, plus 50GB scale for group bys and joins</li>
<li>Median of 3 runs</li>
<li>Using a MacBook Pro M1 with 16GB RAM</li>
<li>DuckDB Versions 0.2.7 through 1.0.0
<ul>
<li>Nearly 3 years, from 2021-06-14 to 2024-06-03</li>
</ul>
</li>
<li>Default settings</li>
<li>Pandas pre-version 0.5.1, Apache Arrow 0.5.1+</li>
</ul>
<h2 id="overall-benchmark-results">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#overall-benchmark-results">Overall Benchmark Results</a>
</h2>
<p>The latest DuckDB can complete one run of the full benchmark suite in under 35 seconds, while version 0.2.7 required nearly 500 seconds for the same task in June 2021.
<strong>That is 14 times faster, in only 3 years!</strong></p>
<h3 id="performance-over-time">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#performance-over-time">Performance over Time</a>
</h3>
<div id="overall_results_by_time" style="width:100%;height:400px;"></div>
<blockquote>
<p>Note These graphs are interactive, thanks to <a href="https://plotly.com/javascript/">Plotly.js</a>!
Feel free to filter the various series (single click to hide, double click to show only that series) and click-and-drag to zoom in.
Individual benchmark results are visible on hover.</p>
</blockquote>
<p>The above plot shows the median runtime in seconds for all tests.
Due to the variety of uses for window functions, and their relative algorithmic complexity, the 16 window function tests require the most time of any category.</p>
<div id="overall_results_by_time_relative" style="width:100%;height:400px;"></div>
<p>This plot normalizes performance to the latest version of DuckDB to show relative improvements over time.
If you look at the point in time when you most recently measured DuckDB performance, that number will show you how many times faster DuckDB is now!</p>
<p>A portion of the overall improvement is DuckDB's addition of multi-threading, which became the default in November 2021 with version 0.3.1.
DuckDB also moved to a push-based execution model in that version for additional gains.
Parallel data loading boosted performance in December 2022 with version 0.6.1, as did improvements to the core <code class="language-plaintext highlighter-rouge">JOIN</code> algorithm.
We will explore other improvements in detail later in the post.</p>
<p>However, we see that all aspects of the system have seen improvements, not just raw query performance!
DuckDB focuses on the entire data analysis workflow, not just aggregate or join performance.
CSV parsing has seen significant gains, import and export have improved significantly, and window functions have improved the most of all.</p>
<p>What was the slight regression from December 2022 to June 2023?
Window functions received additional capabilities and experienced a slight performance degradation in the process.
However, from June 2023 onward we see substantial performance improvement across the board for window functions.
If window functions are filtered out of the chart, we see a smoother trend.</p>
<p>You may also notice that starting with version 0.9 in September 2023, the performance appears to plateau.
What is happening here?
First, don't forget to zoom in!
Over the last year, DuckDB has still improved over 3x!
More recently, the DuckDB Labs team focused on scalability by developing algorithms that support larger-than-memory calculations.
We will see the fruits of those labors in the scale section later on!
In addition, DuckDB focused exclusively on bug fixes in versions 0.10.1, 0.10.2, and 0.10.3 in preparation for an especially robust DuckDB 1.0.
Now that those two major milestones (larger than memory calculations and DuckDB 1.0) have been accomplished, performance improvements will resume!
It is worth noting that the boost from moving to multi-threading will only occur once, but there are still many opportunities moving forward.</p>
<h3 id="performance-by-version">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#performance-by-version">Performance by Version</a>
</h3>
<p>We can also recreate the overall plot by version rather than by time.
This demonstrates that DuckDB has been doing more frequent releases recently.
See <a href="https://duckdb.org/docs/dev/release_calendar.html">DuckDB's release calendar</a> for the full version history.</p>
<div id="overall_results_by_version" style="width:100%;height:400px;"></div>
<p>If you remember the version that you last tested, you can compare how much faster things are now with 1.0!</p>
<div id="overall_results_by_version_relative" style="width:100%;height:400px;"></div>
<h2 id="results-by-workload">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#results-by-workload">Results by Workload</a>
</h2>
<h3 id="csv-reader">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#csv-reader">CSV Reader</a>
</h3>
<div id="perf_over_time_csv_reader_area" style="width:100%;height:400px;"></div>
<p>DuckDB has invested substantially in building a <a href="https://duckdb.org/2023/10/27/csv-sniffer.html">fast and robust CSV parser</a>.
This is often the first task in a data analysis workload, and it tends to be undervalued and underbenchmarked.
DuckDB has <strong>improved CSV reader performance by nearly 3x</strong>, while adding the ability to handle many more CSV dialects automatically.</p>
<h3 id="group-by">
<a style="text-decoration: none;" href="https://duckdb.org/2024/06/26/benchmarks-over-time.html#group-by">Group By</a>
</h3>
<div id="perf_over_time_group_by_area" style="width:100%;height:400px;"></div>
<p>Group by or aggregation operations are critical steps in OLAP workloads, and have therefore received substantial focus in DuckDB, <strong>improving over 12x in the last 3 years</strong>.</p>
<p>In November 2021, version 0.3.1 enabled multithreaded aggregation by default, providing a significant speedup.</p>
<p>In December 2022, data loads into tables were parallelized with the release of version 0.6.1.
This is another example of improving the entire data workflow, as this group by benchmark actually stressed the insertion performance substantially.
|
wonktondI
pushed a commit
to wonktondI/RSSHub
that referenced
this pull request
Jul 28, 2024
* feat(route): add route for DuckDB news * fix(route): fix the missing of full content and author of DuckDB news
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Involved Issue / 该 PR 相关 Issue
Close #
Example for the Proposed Route(s) / 路由地址示例
New RSS Route Checklist / 新 RSS 路由检查表
PuppeteerNote / 说明