Skip to content

[Test] test change #124

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 40 commits into
base: users/yangqiao/AddTriggeringPipeline
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
40 commits
Select commit Hold shift + click to select a range
99c07ce
Avoid including internal libintelmath header. (#99)
mbanck-cd Mar 12, 2025
23e6751
Merged PR 1588069: Set analyze statistics to 0 for unique index uuid …
visridha Feb 26, 2025
6e39f12
Merged PR 1591206: [engine][collation] Collation Support for Query Ma…
CorneliusOsei2 Feb 27, 2025
f200c96
Merged PR 1542893: PostBatchResume token implementation for change st…
rajeshkt78 Feb 27, 2025
746cd8c
Merged PR 1588377: [engine][collation][1/n] Collation Support for Agg…
CorneliusOsei2 Feb 28, 2025
07f6bec
Merged PR 1523428: [schema validation] apply to stage $merge/$out
jiahu2 Feb 28, 2025
dfc690c
Merged PR 1587905: Enabling large unique index keys by default
lucasbfernandes Mar 1, 2025
9849d49
Merged PR 1592153: [engine][collation] Support for Array Aggregation …
CorneliusOsei2 Mar 4, 2025
07871fa
Merged PR 1593971: Add new GUC to support custom index handler op cla…
safern Mar 4, 2025
e679d4f
Merged PR 1588576: [engine][let] Support for Query Match
CorneliusOsei2 Mar 5, 2025
8bf9489
Merged PR 1592819: [engine][collation] Support for redact
CorneliusOsei2 Mar 5, 2025
90c92c4
Merged PR 1591616: [engine][collation] Support for project
CorneliusOsei2 Mar 5, 2025
1341b55
Merged PR 1599232: [engine][collation] Support for Set Operators
CorneliusOsei2 Mar 7, 2025
d1b0992
Merged PR 1591622: [engine][collation] Support for addFields & set
CorneliusOsei2 Mar 10, 2025
b2ef1f3
Merged PR 1585449: [json schema] enable $jsonSchema in query command
jiahu2 Mar 12, 2025
58e9029
Merged PR 1590489: top level let support with $geoNear stage
diipak-bisht Mar 12, 2025
b7606c8
Merged PR 1605600: Fix densify for PG17
diipak-bisht Mar 12, 2025
48326a6
Merged PR 1606155: [Cleanup] Delete old code - Delete default Feature…
visridha Mar 13, 2025
7d31ca1
Merged PR 1605959: [Index Changes] A few fixes for background indexin…
visridha Mar 14, 2025
70887c4
Merged PR 1601128: [engine][collation] Support for replaceRoot
CorneliusOsei2 Mar 16, 2025
dec0c84
Merged PR 1610308: Make range index optimizations only run for RUM in…
safern Mar 17, 2025
bc9a5f7
Merged PR 1610202: [Livesite Fixes] add logging and fix continuation …
visridha Mar 17, 2025
98a5603
Merged PR 1594069: [engine][collation] Support for find
CorneliusOsei2 Mar 18, 2025
b405e3d
Merged PR 1610601: [Cleanup] Code Cleanups on Versions and DocumentDB
visridha Mar 18, 2025
38af796
Merged PR 1612052: [Infra] Delete Helio test files, Reduce index buil…
visridha Mar 19, 2025
9fa289e
DocumentDB sync sha=282647cb8
shuaitian-git Mar 19, 2025
546a2e4
Add dockerfile for pre-build image (#105)
fatmelon Mar 19, 2025
3a369bf
Fix handling of explicit `maxTimeMS` zero values (#41) (#111)
AlekSi Mar 25, 2025
399e01d
Merged PR 1539417: [Operator] $toUUID in Mongo 8.0
xingfan-msft Mar 19, 2025
86ce366
Merged PR 1556142: DateFromString part-2: Add more functionalities to…
Mar 20, 2025
5a8db34
Merged PR 1614043: [Operator] Support extended $getfield for 8.0
xingfan-msft Mar 21, 2025
0c77472
Merged PR 1614124: [Infra] adding support for documentdb_distributed …
LordParag Mar 21, 2025
80df5a1
Merged PR 1604750: [perf][creation_time] Alter creation time : part 1
LordParag Mar 24, 2025
f31e0b6
Merged PR 1615832: Support for pushdown of in to PFE indexes
visridha Mar 24, 2025
580a0a8
Merged PR 1615448: Version update for 102 release
gaurm Mar 24, 2025
b8bb5f3
DocumentDB sync sha=de048377f
shuaitian-git Mar 26, 2025
3f6d186
first commit (#114)
fatmelon Mar 26, 2025
632baa0
update (#115)
fatmelon Mar 26, 2025
06cd6b0
Update CHANGELOG.md for v0.102-0 released (#120)
AvijitkGupta Mar 26, 2025
916e359
test change
Mar 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
126 changes: 97 additions & 29 deletions .github/containers/Build-Ubuntu/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,5 +1,13 @@
FROM --platform=linux/amd64 mcr.microsoft.com/mirror/docker/library/ubuntu:20.04
FROM mcr.microsoft.com/mirror/docker/library/ubuntu:22.04
ARG POSTGRES_INSTALL_ARG=
ARG PG_VERSION=16
ARG CITUS_VERSION=12

# declare installed PG version and Citus version
ENV PG_VERSION=${PG_VERSION}
ENV CITUS_VERSION=${CITUS_VERSION}

# Install build essentials - Compiler, debugger, make, etc.
RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
&& apt-get install -qy \
wget \
@@ -12,6 +20,21 @@ RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
pkg-config \
cmake \
git \
locales \
gcc \
gdb \
libipc-run-perl \
unzip \
apt-transport-https \
bison \
flex \
libreadline-dev \
zlib1g-dev \
libkrb5-dev \
software-properties-common \
libtool \
libicu-dev \
libssl-dev \
&& rm -rf /var/lib/apt/lists/*

# Add pgdg repo
@@ -20,45 +43,90 @@ RUN echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg m
apt-get update && \
rm -rf /var/lib/apt/lists/*

RUN curl -4sSf https://packagecloud.io/install/repositories/citusdata/community/script.deb.sh | bash

RUN apt-get update && DEBIAN_FRONTEND=noninteractive \
&& apt-get install -y \
postgresql-15 \
postgresql-server-dev-15 \
libpq-dev \
postgresql-15-pgcron \
postgresql-15-pgvector \
postgresql-15-postgis-3 \
postgresql-15-rum \
&& rm -rf /var/lib/apt/lists/*

# Install Libbson, decimal128, pcre
# Prepare for running install scripts
ENV CLEANUP_SETUP=1
ENV INSTALL_DEPENDENCIES_ROOT=/tmp/install_setup
RUN mkdir -p /tmp/install_setup
COPY scripts/ /tmp/install_setup
RUN MAKE_PROGRAM=cmake /tmp/install_setup/install_setup_libbson.sh
RUN /tmp/install_setup/install_setup_pcre2.sh
RUN /tmp/install_setup/install_setup_intel_decimal_math_lib.sh
RUN /tmp/install_setup/install_citus_indent.sh

# Copy setup_versions.sh which decides versions of the dependencies to install.
COPY scripts/setup_versions.sh /tmp/install_setup/

# Install libbson
COPY scripts/install_setup_libbson.sh /tmp/install_setup
RUN [ "bin/bash", "-c", "export MAKE_PROGRAM=cmake && /tmp/install_setup/install_setup_libbson.sh" ]

# Copy utils.sh
COPY scripts/utils.sh /tmp/install_setup/

# Install postgres
COPY scripts/install_setup_postgres.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "/tmp/install_setup/install_setup_postgres.sh -d \"/usr/lib/postgresql/${PG_VERSION}\" $POSTGRES_INSTALL_ARG -v ${PG_VERSION}" ]

# Install RUM from source
COPY scripts/install_setup_rum_oss.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_rum_oss.sh" ]

# Install citus
COPY scripts/install_setup_citus_core_oss.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_citus_core_oss.sh ${CITUS_VERSION}" ]

# Install citus-indent
COPY scripts/install_citus_indent.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "/tmp/install_setup/install_citus_indent.sh" ]

# Install SYSTEM_ROWS
COPY scripts/install_setup_system_rows.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_system_rows.sh" ]

# Install PG_CRON
COPY scripts/install_setup_pg_cron.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_pg_cron.sh" ]

# Download Decimal128 Intel library
COPY scripts/install_setup_intel_decimal_math_lib.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "/tmp/install_setup/install_setup_intel_decimal_math_lib.sh" ]

# Download PCRE2 library
COPY scripts/install_setup_pcre2.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "/tmp/install_setup/install_setup_pcre2.sh" ]

# Install PG_VECTOR
COPY scripts/install_setup_pgvector.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_pgvector.sh" ]

# Install PostGIS from source
RUN add-apt-repository -y ppa:ubuntugis/ppa && apt-get update && \
apt-get install -qy \
libproj-dev \
libxml2-dev \
libjson-c-dev \
libgeos-dev \
libgeos++-dev \
&& rm -rf /var/lib/apt/lists/*
COPY scripts/install_setup_postgis.sh /tmp/install_setup/
RUN [ "bin/bash", "-c", "PGVERSION=$PG_VERSION /tmp/install_setup/install_setup_postgis.sh" ]

# locale
RUN rm -rf /var/lib/apt/lists/* \
&& localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8

ENV LANG en_US.utf8
ENV LANG en_US.UTF-8
ENV LC_ALL en_US.UTF-8
ENV LANGUAGE en_US

# Create documentdb user
RUN useradd -ms /bin/bash documentdb -G sudo
RUN echo "%sudo ALL=(ALL:ALL) NOPASSWD: ALL" >> /etc/sudoers.d/no-pass-ask

RUN adduser --disabled-password --gecos "" --uid 1001 runner \
&& groupadd docker --gid 123 \
&& usermod -aG sudo runner \
&& usermod -aG docker runner \
&& echo "%sudo ALL=(ALL:ALL) NOPASSWD:ALL" > /etc/sudoers \
&& echo "Defaults env_keep += \"DEBIAN_FRONTEND\"" >> /etc/sudoers
# Add postgres to path for sudo commands.
# Add path for sudoers
RUN cat /etc/sudoers | grep secure_path | sed "s/\:\/bin\:/\:\/bin\:\/usr\/lib\/postgresql\/$PG_VERSION\/bin\:/" >> /etc/sudoers.d/postgres_path

WORKDIR /home/runner
# Add PG to the path
ENV PATH=$PATH:/usr/lib/postgresql/$PG_VERSION/bin

USER runner
USER documentdb
WORKDIR /home/documentdb

LABEL org.opencontainers.image.source=https://github.com/microsoft/documentdb
LABEL org.opencontainers.image.description="DocumentDB ubuntu build image"
11 changes: 11 additions & 0 deletions .github/containers/Build-Ubuntu/Dockerfile_prebuild
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
FROM mcr.microsoft.com/cosmosdb/ubuntu/documentdb-oss:base

WORKDIR /home/documentdb/code

COPY . /home/documentdb/code

RUN git config --global --add safe.directory /home/DocumentDB/code
RUN make && sudo make install

# Set the entry point to run the new script at container startup
CMD ["bash", "-c", "/home/documentdb/code/scripts/start_oss_server.sh & tail -f /dev/null"]
17 changes: 15 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,11 +1,24 @@
### documentdb v0.102-0 (Unreleased) ###
### documentdb v0.103-0 (Unreleased) ###

### documentdb v0.102-0 (March 26, 2025) ###
* Support index pushdown for vector search queries *[Bugfix]*
* Support exact search for vector search queries *[Feature]*
* Inline $match with let in $lookup pipelines as JOIN Filter *[Perf]*
* Support TTL indexes *[Bugfix]* (#34)
* Support joining between postgres and documentdb tables *[Feature]* (#61)
* Support current_op command *[Feature]* (#59)
* Support for list_databases command *[Feature]* (#45)
* Disable analyze statistics for unique index uuid columns which improves resource usage *[Perf]*
* Support collation with `$expr`, `$in`, `$cmp`, `$eq`, `$ne`, `$lt`, `$lte`, `$gt`, `$gte` comparison operators (Opt-in) *[Feature]*
* Support collation in `find`, aggregation `$project`, `$redact`, `$set`, `$addFields`, `$replaceRoot` stages (Opt-in) *[Feature]*
* Support collation with `$setEquals`, `$setUnion`, `$setIntersection`, `$setDifference`, `$setIsSubet` in the aggregation pipeline (Opt-in) *[Feature]*
* Support unique index truncation by default with new operator class *[Feature]*
* Top level aggregate command `let` variables support for `$geoNear` stage *[Feature]*
* Enable Backend Command support for Statement Timeout *[Feature]*
* Support type aggregation operator `$toUUID`. *[Feature]*
* Support Partial filter pushdown for `$in` predicates *[Perf]*
* Support the $dateFromString operator with full functionality *[Feature]*
* Support extended syntax for `$getField` aggregation operator. Now the value of 'field' could be an expression that resolves to a string. *[Feature]*

### documentdb v0.101-0 (February 12, 2025) ###
* Push $graphlookup recursive CTE JOIN filters to index *[Perf]*
@@ -16,4 +29,4 @@
* Fix Merge stage not outputing to target collection *[Bugfix]* (#20)

### documentdb v0.100-0 (January 23rd, 2025) ###
Initial Release
Initial Release
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
1
# Introduction

`DocumentDB` is the engine powering vCore-based Azure Cosmos DB for MongoDB. It offers a native implementation of document-oriented NoSQL database, enabling seamless CRUD operations on BSON data types within a PostgreSQL framework. Beyond basic operations, DocumentDB empowers you to execute complex workloads, including full-text searches, geospatial queries, and vector embeddings on your dataset, delivering robust functionality and flexibility for diverse data management needs.
@@ -69,7 +70,7 @@ Step 4: Build & Deploy the binaries
make
```

Note: Run in case of an unsuccessful build `git config --global --add safe.directory /home/DocumentDB/code` within image.
Note: Run in case of an unsuccessful build `git config --global --add safe.directory /home/documentdb/code` within image.

```bash
sudo make install
6 changes: 5 additions & 1 deletion documentdb_errors.csv
Original file line number Diff line number Diff line change
@@ -504,4 +504,8 @@ OrdinalPosition,ErrorName
503,Location7750302
504,Location7750303
505,Location51003
506,Location7749501
506,Location7749501
507,Location5860402
508,Location7555701
509,Location7555702
510,Location3041704
6 changes: 5 additions & 1 deletion error_mappings.csv
Original file line number Diff line number Diff line change
@@ -400,6 +400,7 @@ ErrorMapping,ErrorName
3041701,Location3041701
3041702,Location3041702
3041703,Location3041703
3041704,Location3041704
4031700,IntermediateResultTooLarge
4161100,DollarSetFieldRequiresObject
4161101,DollarSetFieldUnknownArgument
@@ -487,6 +488,7 @@ ErrorMapping,ErrorName
5788200,Location5788200
5788604,Location5788604
5858203,Location5858203
5860402,Location5860402
5876900,Location5876900
5897900,Location5897900
5946802,Location5946802
@@ -500,8 +502,10 @@ ErrorMapping,ErrorName
6586400,Location6586400
7429703,Location7429703
7436100,Location7436100
7555701,Location7555701
7555702,Location7555702
7749501,Location7749501
7750301,Location7750301
7750302,Location7750302
7750303,Location7750303
8993000,Location8993000
8993000,Location8993000
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
comment = 'Multi-Node API surface for DocumentDB'
default_version = '0.102-0'
default_version = '0.103-0'
module_pathname = '$libdir/pg_documentdb_distributed'
relocatable = false
superuser = true
Empty file.
Original file line number Diff line number Diff line change
@@ -26,6 +26,7 @@
#include "utils/version_utils.h"
#include "utils/error_utils.h"
#include "utils/version_utils_private.h"
#include "utils/data_table_utils.h"
#include "api_hooks.h"

extern int MaxNumActiveUsersIndexBuilds;
@@ -72,7 +73,7 @@ static void ParseVersionString(ExtensionVersion *extensionVersion, char *version
static bool SetupCluster(bool isInitialize);
static void SetPermissionsForReadOnlyRole(void);
static void CheckAndReplicateReferenceTable(const char *schema, const char *tableName);
static ArrayType * GetCollectionIds(void);


PG_FUNCTION_INFO_V1(command_initialize_cluster);
PG_FUNCTION_INFO_V1(command_complete_upgrade);
@@ -321,6 +322,11 @@ SetupCluster(bool isInitialize)
CheckAndReplicateReferenceTable(ApiDistributedSchemaName, relationName->data);
}

if (ShouldRunSetupForVersion(&versions, DocDB_V0, 101, 0))
{
AlterCreationTime();
}

/* we call the post setup cluster hook to allow the extension to do any additional setup */
PostSetupClusterHook(isInitialize, &ShouldRunSetupForVersionForHook, &versions);

@@ -814,23 +820,23 @@ DropLegacyChangeStream()
Datum *elements = NULL;
int numElements = 0;
bool *val_is_null_marker;
deconstruct_array(arrayValue, INT4OID, sizeof(int), true, TYPALIGN_INT,
deconstruct_array(arrayValue, INT8OID, sizeof(int64), true, TYPALIGN_INT,
&elements, &val_is_null_marker, &numElements);

for (int i = 0; i < numElements; i++)
{
int collection_id = DatumGetInt32(elements[i]);
int64_t collection_id = DatumGetInt64(elements[i]);

resetStringInfo(cmdStr);
appendStringInfo(cmdStr,
"ALTER TABLE IF EXISTS %s.documents_%d DROP COLUMN IF EXISTS change_description;",
"ALTER TABLE IF EXISTS %s.documents_%ld DROP COLUMN IF EXISTS change_description;",
ApiDataSchemaName, collection_id);
ExtensionExecuteQueryViaSPI(cmdStr->data, readOnly, SPI_OK_UTILITY,
&isNull);

resetStringInfo(cmdStr);
appendStringInfo(cmdStr,
"DROP TRIGGER IF EXISTS record_changes_trigger ON %s.documents_%d;",
"DROP TRIGGER IF EXISTS record_changes_trigger ON %s.documents_%ld;",
ApiDataSchemaName, collection_id);
ExtensionExecuteQueryViaSPI(cmdStr->data, readOnly, SPI_OK_UTILITY,
&isNull);
@@ -988,15 +994,15 @@ SetPermissionsForReadOnlyRole()
Datum *elements = NULL;
int numElements = 0;
bool *val_is_null_marker;
deconstruct_array(arrayValue, INT4OID, sizeof(int), true, TYPALIGN_INT,
deconstruct_array(arrayValue, INT8OID, sizeof(int64), true, TYPALIGN_INT,
&elements, &val_is_null_marker, &numElements);

for (int i = 0; i < numElements; i++)
{
int collection_id = DatumGetInt32(elements[i]);
int64_t collection_id = DatumGetInt64(elements[i]);
resetStringInfo(cmdStr);
appendStringInfo(cmdStr,
"GRANT SELECT ON %s.documents_%d TO %s;",
"GRANT SELECT ON %s.documents_%ld TO %s;",
ApiDataSchemaName, collection_id, ApiReadOnlyRole);
ExtensionExecuteQueryViaSPI(cmdStr->data, readOnly, SPI_OK_UTILITY,
&isNull);
@@ -1037,30 +1043,6 @@ CheckAndReplicateReferenceTable(const char *schema, const char *tableName)
}


/*
* Gets the collection Ids where view_definition is NULL
*/
static ArrayType *
GetCollectionIds()
{
bool isNull = false;
bool readOnly = true;
StringInfo cmdStr = makeStringInfo();
appendStringInfo(cmdStr,
"SELECT array_agg(DISTINCT collection_id)::int4[] FROM %s.collections where view_definition IS NULL;",
ApiCatalogSchemaName);
Datum versionDatum = ExtensionExecuteQueryViaSPI(cmdStr->data, readOnly,
SPI_OK_SELECT, &isNull);

if (isNull)
{
return NULL;
}

return DatumGetArrayTypeP(versionDatum);
}


static char *
UpdateClusterMetadata(bool isInitialize)
{
Original file line number Diff line number Diff line change
@@ -596,6 +596,7 @@ InitializeDocumentDBDistributedHooks(void)
get_distributed_application_name_hook = GetDistributedApplicationNameCore;
ensure_metadata_table_replicated_hook = EnsureMetadataTableReplicatedCore;
DefaultInlineWriteOperations = false;
ShouldUpgradeDataTables = false;
UpdateColocationHooks();

try_get_extended_version_refresh_query_hook = TryGetExtendedVersionRefreshQueryCore;
Original file line number Diff line number Diff line change
@@ -99,6 +99,7 @@ test: bson_aggregation_stage_merge_tests commands_create_indexes_text bson_aggre
test: bson_aggregation_pipeline_tests_geonear bson_aggregation_pipeline_stage_setWindowFields

test: cursors_basic_support cursors_seqscan bson_aggregation_cursor_tests
test: commands_create_unique_index_stats

# Not to be run concurrently with other tests to prevent flakiness
test: mongo_feature_counters
Loading
Oops, something went wrong.
Loading
Oops, something went wrong.