feat: Add subquery support in pipeline by cherylEnkidu · Pull Request #2323 · googleapis/java-firestore

cherylEnkidu · 2026-02-24T19:09:57Z

feat: add support for Pipeline subqueries

This PR introduces support for Pipeline Subqueries, allowing users to perform complex data transformations and aggregations on subcollections within a pipeline.

Key Features

Subcollection Pipelines: Added Pipeline.subcollection(String path) to initiate a pipeline on a subcollection of the current document.
Variable Definition: Added Pipeline.define(...) to bind values to variables for reuse within the pipeline scope.
Expression Enhancements:
- Expression.currentDocument(): Reference the current document in a pipeline stage.
- Expression.variable(String name): Access defined variables.
- Pipeline.toArrayExpression() and Pipeline.toScalarExpression(): Convert subquery results into array or scalar expressions for use in parent pipelines.

Testing

Added ITPipelineSubqueryTest.java containing integration tests for subquery functionality.
Verified compatibility with the Enterprise backend.

generated-files-bot · 2026-02-24T19:10:01Z

Warning: This pull request is touching the following templated files:

proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/DocumentProto.java
proto-google-cloud-firestore-v1/src/main/java/com/google/firestore/v1/Value.java

dlarocque · 2026-03-25T17:37:13Z

+  @InternalApi
+  public FunctionExpression(String name, List<? extends Expression> params) {


Why do we make this internal?

dlarocque · 2026-03-25T17:48:42Z

+    bookDocs =
+        ImmutableMap.<String, Map<String, Object>>builder()
+            .put(
+                "book1",
+                ImmutableMap.<String, Object>builder()
+                    .put("title", "The Hitchhiker's Guide to the Galaxy")
+                    .put("author", "Douglas Adams")
+                    .put("genre", "Science Fiction")
+                    .put("published", 1979)


It looks like this is a copy of the bookDocs from the ITPipelineTest.java setup. Can you pull this setup code out from both of these places and put it into a function that we can reuse in both places? Maybe we could remove ~150 LOC from here.

I am skeptical about consolidating test setup here. While it serves to remove some duplication, we are talking about tests here, and having all test data ready in the same file has benefits to both human and agents.

I do want to see the multiple review collections we setup in the individual tests to be moved in here. That is, we setup both the books, and reviews or some other collections here. The number of documents can be shrinked, like 3,4 books and their reviews might be enough.

Thanks for the feedback! Here are my thoughts:

Cross-SDK Consistency: I'd prefer to keep this doc data shared. The web SDKs reuse the same dataset between the SubqueryTest and the ITPipelineTest. Sharing it here ensures our datasets remain consistent across all SDKs.

Future Improvements: I agree we can keep an eye on this. If we find later that we need different types of datasets, or if we want to save on document creation, we can always revisit this and separate the data in the future.

wu-hui · 2026-03-25T17:54:06Z

@@ -0,0 +1,36 @@
+/*
+ * Copyright 2024 Google LLC


wu-hui · 2026-03-25T17:54:29Z

@@ -0,0 +1,39 @@
+/*
+ * Copyright 2025 Google LLC


wu-hui · 2026-03-26T15:12:22Z

+    bookDocs =
+        ImmutableMap.<String, Map<String, Object>>builder()
+            .put(
+                "book1",
+                ImmutableMap.<String, Object>builder()
+                    .put("title", "The Hitchhiker's Guide to the Galaxy")
+                    .put("author", "Douglas Adams")
+                    .put("genre", "Science Fiction")
+                    .put("published", 1979)


I am skeptical about consolidating test setup here. While it serves to remove some duplication, we are talking about tests here, and having all test data ready in the same file has benefits to both human and agents.

I do want to see the multiple review collections we setup in the individual tests to be moved in here. That is, we setup both the books, and reviews or some other collections here. The number of documents can be shrinked, like 3,4 books and their reviews might be enough.

dlarocque · 2026-03-30T17:59:49Z

+   * @param key The key of the field to access.
+   * @return An {@link Expression} representing the value of the field.
+   */
+  @BetaApi


i'm assuming these beta annotations will be removed in a later PR

I will remove these in this PR :)

…query

**Overview** This PR introduces support for pipeline subqueries, variable definitions, and joins in the Go Firestore SDK, achieving strict 1:1 feature parity with the Java SDK's pipeline APIs googleapis/java-firestore#2323. **What are Subqueries?** In Firestore pipelines, subqueries allow you to embed an entire pipeline execution as a value within a single stage of an outer pipeline. This is incredibly powerful for performing complex "join-like" operations across different collections. For example, while querying a restaurants collection, you can use a subquery to fetch, filter, and aggregate all documents from a nested reviews subcollection, and embed that aggregated result (e.g., average_rating) directly into the restaurant document being returned. Subqueries can be evaluated into either an array of results (ToArrayExpression()) or a single scalar value (ToScalarExpression()). **Key Features & API Additions:** * **Subqueries (Joins):** * Implemented the Subcollection(path string) package-level function to instantiate relative-scope pipelines. * Added ToScalarExpression() and ToArrayExpression() methods on Pipeline to explicitly convert subqueries into expressions, allowing them to be seamlessly embedded inside stages like AddFields and Where. * **Variable Definition & References:** * Introduced the Define pipeline stage and the AliasedExpressions variadic helper to ergonomically bind values to variables. * Added Variable("name") and CurrentDocument() top-level functions to reference bound values in subsequent pipeline stages. * **Field Access & Overloading:** * Implemented GetField (accepting any to support both string and Expression arguments, mirroring Java's method overloading). **Type Safety & Defensive Constraints:** * Strict Aliasing: Expression.As() now explicitly returns *AliasedExpression (similar to Java: https://github.com/googleapis/java-firestore/blob/0c8188520dfbada0d3fef0719e4f95fc231306be/google-cloud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java#L6608-L6621) rather than a generic Selectable interface, and the Define stage strictly requires []*AliasedExpression. * Pipeline Scope Validation: Added validation to ensure relative-scope pipelines (e.g., those created via Subcollection) cannot be executed directly or passed into a Union() stage. Attempts to do so now return descriptive errors identical to the Java SDK's IllegalStateException and IllegalArgumentException.

cherylEnkidu added 5 commits February 20, 2026 17:48

save work

c02ef30

Fix breaking tests

b65de3a

Fix testSingleLookupScalarSubquery

8a4526f

Fix broken tests

097cdd5

change syntax for define and return type of as function

ed82cd4

cherylEnkidu requested review from a team as code owners February 24, 2026 19:09

product-auto-label Bot added size: l Pull request size is large. api: firestore Issues related to the googleapis/java-firestore API. labels Feb 24, 2026

add documentation and overload for getField()

7acdb11

cherylEnkidu force-pushed the cheryl/subquery branch from 46ed5ab to 7acdb11 Compare February 25, 2026 22:09

product-auto-label Bot added size: xl Pull request size is extra large. size: l Pull request size is large. and removed size: l Pull request size is large. size: xl Pull request size is extra large. labels Feb 25, 2026

change proto

d0f184a

cherylEnkidu marked this pull request as draft February 26, 2026 19:33

cherylEnkidu force-pushed the cheryl/subquery branch from d504667 to d0f184a Compare February 26, 2026 19:34

product-auto-label Bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 26, 2026

chore: generate libraries at Thu Feb 26 19:35:06 UTC 2026

cdb7f8a

product-auto-label Bot added size: l Pull request size is large. and removed size: xl Pull request size is extra large. labels Feb 26, 2026

format code

6af735f

product-auto-label Bot added size: xl Pull request size is extra large. and removed size: l Pull request size is large. labels Feb 26, 2026

cherylEnkidu added 3 commits February 26, 2026 16:19

add tests for toArrayExpression

2efa8ec

move the subquery test to independent file

c5bf66c

Add tests for subcollection

6208313

cherylEnkidu commented Feb 27, 2026

View reviewed changes

Comment thread ...loud-firestore/src/main/java/com/google/cloud/firestore/pipeline/expressions/Expression.java

cherylEnkidu added 4 commits February 27, 2026 16:01

change the test

0bd670d

Test subcollection used within Union stage

d7b6c69

replace mapGet with getField

ce2b307

support subcollection in test

048f664

cherylEnkidu marked this pull request as ready for review March 11, 2026 16:52

cherylEnkidu and others added 6 commits March 11, 2026 14:14

Merge branch 'main' into cheryl/subquery

940848b

chore: generate libraries at Wed Mar 11 18:14:28 UTC 2026

36fb820

improve error message

b69c5e8

merge in main

11ee929

chore: generate libraries at Tue Mar 24 17:04:45 UTC 2026

0ea96aa

add decode support

913d44f

cherylEnkidu changed the title ~~Subquery~~ feat: Add subquery support in pipeline Mar 24, 2026

Make the subquery test only run in nightly environment

ac994ac

cherylEnkidu requested a review from dlarocque March 25, 2026 17:11

cherylEnkidu assigned dlarocque Mar 25, 2026

dlarocque reviewed Mar 25, 2026

View reviewed changes

wu-hui reviewed Mar 26, 2026

View reviewed changes

cherylEnkidu and others added 3 commits March 27, 2026 16:50

Merge main into current branch

6f90616

address feedbacks

52d2c2b

chore: generate libraries at Fri Mar 27 22:19:12 UTC 2026

67cb5dc

dlarocque approved these changes Mar 30, 2026

View reviewed changes

cherylEnkidu added 3 commits March 30, 2026 15:23

remove beta annotation

9443a4a

Merge remote-tracking branch 'origin/cheryl/subquery' into cheryl/sub…

8400c47

…query

Resolve merge conflicts in Pipeline and PipelineSource

727c783

cherylEnkidu merged commit cbf87ac into main Mar 31, 2026
28 checks passed

cherylEnkidu deleted the cheryl/subquery branch March 31, 2026 21:15

release-please Bot mentioned this pull request Mar 31, 2026

chore(main): release 3.40.0 #2345

Merged

bhshkh mentioned this pull request Apr 6, 2026

feat(firestore): pipeline subqueries googleapis/google-cloud-go#14365

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add subquery support in pipeline#2323

feat: Add subquery support in pipeline#2323
cherylEnkidu merged 33 commits intomainfrom
cheryl/subquery

cherylEnkidu commented Feb 24, 2026 •

edited

Loading

Uh oh!

generated-files-bot Bot commented Feb 24, 2026 •

edited

Loading

Uh oh!

Uh oh!

dlarocque Mar 25, 2026

Uh oh!

dlarocque Mar 25, 2026

Uh oh!

wu-hui Mar 26, 2026 •

edited

Loading

Uh oh!

cherylEnkidu Mar 27, 2026

Uh oh!

wu-hui Mar 25, 2026

Uh oh!

wu-hui Mar 25, 2026

Uh oh!

wu-hui Mar 26, 2026 •

edited

Loading

Uh oh!

dlarocque Mar 30, 2026

Uh oh!

cherylEnkidu Mar 30, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		@InternalApi
		public FunctionExpression(String name, List<? extends Expression> params) {

Conversation

cherylEnkidu commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Key Features

Testing

Uh oh!

generated-files-bot Bot commented Feb 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

dlarocque Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

dlarocque Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

wu-hui Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cherylEnkidu Mar 27, 2026

Choose a reason for hiding this comment

Uh oh!

wu-hui Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

wu-hui Mar 25, 2026

Choose a reason for hiding this comment

Uh oh!

wu-hui Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dlarocque Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

cherylEnkidu Mar 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cherylEnkidu commented Feb 24, 2026 •

edited

Loading

generated-files-bot Bot commented Feb 24, 2026 •

edited

Loading

wu-hui Mar 26, 2026 •

edited

Loading

wu-hui Mar 26, 2026 •

edited

Loading