Skip to content

2026-04-13 Custom DC stable release#6201

Merged
gmechali merged 24 commits intodatacommonsorg:customdc_stablefrom
gmechali:cdcStableUpdate
Apr 13, 2026
Merged

2026-04-13 Custom DC stable release#6201
gmechali merged 24 commits intodatacommonsorg:customdc_stablefrom
gmechali:cdcStableUpdate

Conversation

@gmechali
Copy link
Copy Markdown
Contributor

Highlights

Resources - REMOVE BEFORE MERGING

shixiao-coder and others added 24 commits April 2, 2026 14:28
This PR removes the codepaths that supported the old, un-revamped
/tools/map UI.

# Summary

Now that the "standardized_vis_tool" experience is the default
experience, this PR removes the old /tools/map experience so that we can
eventually delete the flag. In order to keep changes concise, I've only
removed the old /tools/map. The /tools/scatter, /tools/timeline, and
/tools/visualization will come in a separate PR.

# Testing Strategy

Locally, I set the "standardized_vis_tool" flag to False, and verified
that:
* navigating to localhost:8080/tools/map still renders the new,
standardized experience.
* webdriver, client, and python tests still pass
This PR removes the codepaths that supported the old, un-revamped
/tools/timeline UI.

# Summary
Now that the "standardized_vis_tool" experience is the default
experience, this PR removes the old /tools/timeline experience so that
we can eventually delete the flag. In order to keep changes concise,
I've only removed the old /tools/timeline in this PR. Removing
/tools/visualization will come in a separate PR.

Related PRs:
/tools/map is being removed in datacommonsorg#6165
/tools/scatter is being removed in datacommonsorg#6167 

# Testing Strategy
Locally, I set the "standardized_vis_tool" flag to False, and verified
that:

* navigating to localhost:8080/tools/timeline still renders the new,
standardized experience.
* webdriver, client, and python tests still pass
## Description

The webdriver tests recordings were out-of-date and notably did not
contain recordings for the metadata/facet endpoints (which are heavy on
mixer).

This PR

* Updates the tests for data drift
* Adds hashing and fallback entries for the metadata/facet endpoint
* Regenerates the recordings.

## Notes

The custom DC tests in some cases currently expect slightly different
data (or forms of the same data) than the primary tests. Some tests were
updated to make them more flexible.
This PR removes the codepaths that supported the old, un-revamped
/tools/scatter UI.

# Summary
Now that the "standardized_vis_tool" experience is the default
experience, this PR removes the old /tools/scatter experience so that we
can eventually delete the flag. In order to keep changes concise, I've
only removed the old /tools/scatter in this PR. Removing
/tools/timeline, and /tools/visualization will come in a separate PR.

Related PR:
/tools/map is being removed in datacommonsorg#6165 

# Testing Strategy
Locally, I set the "standardized_vis_tool" flag to False, and verified
that:

* navigating to localhost:8080/tools/scatter still renders the new,
standardized experience.
* webdriver, client, and python tests still pass

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
## Issue

[b/493339134](https://b.corp.google.com/issues/493339134)

## Description

This PR converts the usage of ` /v1/variable/search` to `v2` endpoints
centered around `/v2/resolve`.

## Notes

In order to keep search functionality for custom DCs while migrating
away from the `v1`-centric search performed before, we have moved the
logic of the search to use `v2/resolve`.

The `search_statvar` endpoint now follows the following logic:

* If Vertex AI is active (governed by a flag), we use Vertex AI (this
functionality has not changed in this PR)
* Else, if the NL server is running and we are a custom DC, we use the
new `v2/resolve`-centric method to do the search
* Else, we do not allow search (noting that this should not occur in
practice, given the above paths).

When we are in the `v2/resolve` branch, we follow the following logic:

* Resolve Query to Candidate DCIDS: We call `/v2/resolve` to convert the
natural language query into a list of candidate DCIDs.
* Check Observations: If the request filters by entities, we call
`/v2/observation` for all candidates. We discards any variables that
don't have active observation data for those entities (we skip this if
no entities given)
* Filter by typeOf: We calls `/v2/node` with property: "->typeOf" to
ensure candidates are actually StatisticalVariables and filters out any
StatVarGroups or Topics.
* Fetch Names: For the remaining valid DCIDs, we call dc.v2node with
property: "->name" to retrieve their human-readable display names.
* Format Response: Finally, we merges the DCIDs and display names into a
standard result structure ({"statVars": [...], "matches": [],
"statVarGroups": []}) to maintain compatibility with existing frontend
usage.


## Testing

[Stat Var Explorer](http://localhost:8080/tools/statvar)

* When running the server normally: `./run_server.sh -m`, a search will
use Vertex AI and should be identical to what it was before.
* When running the server as a custom DC: `./run_server.sh -m -e
custom`, the new search path is invoked. The search experience should
generally be the same as in master, but the results may not (because we
are using the embeddings-based based `v2/resolve` search that will not
surface all variables.
* When running the server as a custom DC without the NL server:
`./run_server.sh -m -e custom`, not search box should appear in the
front end.

[Timeline Explorer](http://localhost:8080/tools/timeline)

All below are for running as a custom DC `./run_server.sh -m -e custom`

* When no place is selected, a search for population will bring up many
stat vars (a set of results unfiltered by entity)
* When a place is selected (such as India), the same search should
produce noticeably fewer results (only results that are relevant for
India).
This PR removes /tools/visualization, leaving only the redirect to the
new standardized /tools/map, /tools/scatter, /tools/timeline experience.

## Description

This PR cleans up the legacy `/tools/visualization` route and its
associated UI components. It converts the page into a lightweight
client-side redirector that sends users to the standardized
`/tools/map`, `/tools/scatter`, or `/tools/timeline` pages based on the
URL hash parameters.

This is the first part of the cleanup, focusing **only** on the
tools/visualization path. Updates to the individual tool pages and their
tests will be handled in separate atomic PRs.


## Changes
### 🔧 Server-side Routing & Templates
- **`server/routes/tools/html.py`**
- Simplified the `/tools/visualization` route handler to render the
template without server-side data fetching.
- **`server/templates/tools/visualization.html`**
  - Stripped unused styles and Maps API integrations.
### ⚛️ Client-side App & Redirection
- **`static/js/apps/visualization/main.ts`**
- Updated to perform immediate client-side redirection based on the URL
hash.
- **[DELETE] `static/js/apps/visualization/app.tsx`** and associated UI
components (except for redirect utilities).
- **[DELETE] `static/css/tools/visualization.scss`** and removed its
entry from `webpack.config.js`.
### 🛠️ Shared Utilities
- **`static/js/utils/app/visualization_utils.tsx`**
- Decoupled from the deleted `app_context.ts` by defining
`ContextStatVar` and `DisplayOptions` interfaces locally. This unblocks
chart tiles that still share these utilities without pulling in old
visualization app context.


## Testing Strategy
### Manual Verification
1. Navigate to `/tools/visualization#visType=timeline` (or map/scatter)
and verify that it immediately redirects to the new `/tools/timeline`
page (or map/scatter) with the hash parameters intact.
2. Verify that the build completes successfully without `app_context`
missing errors.
3. Verify that even with standardizd_vis_tool flag set to False,
redirect still happens as intended
4. Verify that webdriver, python, and client tests pass locally.
This PR automatically updates the `mixer` and `import` submodules to
their latest `master` branches.

Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
## Description

This PR sets `use_v2_api` to true in local, autopush and dev.
…commonsorg#6147)

Adds a optional "show next" feature to the ranking tile, allowing users
to incrementally load additional entries (e.g. top 100 -> top 200), and
adds this functionality to the new ranking pages.

# Summary
This PR adds an optional button to the ranking tile that when clicked,
will expand the ranking tile table to show more entries. The number of
entries to add on is configured via the new optional `showNextCount`
argument. If showNextCount is 0 or not provided, the button is hidden
and pagination is disabled.

Features / UI Considerations:
* The button is being added in the bottom right corner, where the chart
footer is.
* If both highest and lowest tables are being shown in the tile, each
table gets its own button, and their pagination act independently of
each other.
* A slide down animation has been added for when the user clicks on the
button to make the transition less jarring
* This feature has also been enabled on the web components via a new
`showNextCount` html attribute
* This feature is available on chart configs via the `showNextCount`
RankingTileSpec field.
* Updated the dev subject page at /dev/subject-page-tiles to include all
variants of the ranking tile.

Engineering Considerations:
* This button is NOT being added to the explore page or topic pages, so
there is no expected visual or functional changes to those pages. This
button is only being used on the new Ranking pages.
* Because the button is an icon only, it does not require i18n strings
* Because the ranking metadata used by the subject page configs is
intended to follow protobuf conventions, the callback for updating the
state of the ranking chart can't be passed in via config. In order to
avoid ranking-specific prop drilling through the subject page react
components, I've elected to use a ranking page context instead.

Note:
This PR is the final PR needed for implementation of the new ranking
pages. However, in order to prioritize completing the v0/v1 migration as
fast as possible, I'll prioritize removing the ranking pages, and come
back to add testing for the new ranking page UI and this new button
functionality in a followup PR.

# Left for future PRs

* Updating the web-component documentation on the docsite repo to
include the new `showNextCount` attribute.
* Adding tests for the new button

# Screenshots
### New ranking page UI
<img width="3226" height="5846"
alt="localhost_8080_ranking_Count_Person_County_country_USA (1)"
src="https://github.com/user-attachments/assets/febc6599-b54f-4526-b86f-ca3f8b3d5b86"
/>


### Dev page with ranking tiles with and without pagination
<img width="2560" height="1328" alt="image"
src="https://github.com/user-attachments/assets/a8af7557-c060-4db1-ae41-41dda85633c0"
/>

---------

Co-authored-by: Nick Blumberg <nblumberg@google.com>
## Description

The pinned version of `google-adk` had a security vulnerability. This PR
upgrades that library, along with other libraries that needed to be
upgrade as part of the dependency cascade.

## Notes

This resulted in a number of dependency upgrades (visible in the updated
`pyproject.toml` file. I upgraded the minimal number of libraries
possible to allow us to update `google-adk`.

One of the updates `setuptools` being dropped as the dependency tree.
This is explicitly restored at a version that allows us to run our test
suites.
Bumps [setuptools](https://github.com/pypa/setuptools) from 66.1.1 to
78.1.1.
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pypa/setuptools/blob/main/NEWS.rst">setuptools's
changelog</a>.</em></p>
<blockquote>
<h1>v78.1.1</h1>
<h2>Bugfixes</h2>
<ul>
<li>More fully sanitized the filename in PackageIndex._download. (<a
href="https://redirect.github.com/pypa/setuptools/issues/4946">#4946</a>)</li>
</ul>
<h1>v78.1.0</h1>
<h2>Features</h2>
<ul>
<li>Restore access to _get_vc_env with a warning. (<a
href="https://redirect.github.com/pypa/setuptools/issues/4874">#4874</a>)</li>
</ul>
<h1>v78.0.2</h1>
<h2>Bugfixes</h2>
<ul>
<li>Postponed removals of deprecated dash-separated and uppercase fields
in <code>setup.cfg</code>.
All packages with deprecated configurations are advised to move before
2026. (<a
href="https://redirect.github.com/pypa/setuptools/issues/4911">#4911</a>)</li>
</ul>
<h1>v78.0.1</h1>
<h2>Misc</h2>
<ul>
<li><a
href="https://redirect.github.com/pypa/setuptools/issues/4909">#4909</a></li>
</ul>
<h1>v78.0.0</h1>
<h2>Bugfixes</h2>
<ul>
<li>Reverted distutils changes that broke the monkey patching of command
classes. (<a
href="https://redirect.github.com/pypa/setuptools/issues/4902">#4902</a>)</li>
</ul>
<h2>Deprecations and Removals</h2>
<ul>
<li>Setuptools no longer accepts options containing uppercase or dash
characters in <code>setup.cfg</code>.</li>
</ul>
<!-- raw HTML omitted -->
</blockquote>
<p>... (truncated)</p>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pypa/setuptools/commit/8e4868a036b7fae3208d16cb4e5fe6d63c3752df"><code>8e4868a</code></a>
Bump version: 78.1.0 → 78.1.1</li>
<li><a
href="https://github.com/pypa/setuptools/commit/100e9a61ad24d5a147ada57357425a8d40626d09"><code>100e9a6</code></a>
Merge pull request <a
href="https://redirect.github.com/pypa/setuptools/issues/4951">#4951</a></li>
<li><a
href="https://github.com/pypa/setuptools/commit/8faf1d7e0ca309983252e4f21837b73ee12e960f"><code>8faf1d7</code></a>
Add news fragment.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/2ca4a9fe4758fcd39d771d3d3a5b4840aacebdf7"><code>2ca4a9f</code></a>
Rely on re.sub to perform the decision in one expression.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/e409e8002932f2b86aae7b1abc8f8c2ebf96df2c"><code>e409e80</code></a>
Extract _sanitize method for sanitizing the filename.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/250a6d17978f9f6ac3ac887091f2d32886fbbb0b"><code>250a6d1</code></a>
Add a check to ensure the name resolves relative to the tmpdir.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/d8390feaa99091d1ba9626bec0e4ba7072fc507a"><code>d8390fe</code></a>
Extract _resolve_download_filename with test.</li>
<li><a
href="https://github.com/pypa/setuptools/commit/4e1e89392de5cb405e7844cdc8b20fc2755dbaba"><code>4e1e893</code></a>
Merge <a
href="https://github.com/jaraco/skeleton">https://github.com/jaraco/skeleton</a></li>
<li><a
href="https://github.com/pypa/setuptools/commit/3a3144f0d2887fa37c06550f42a101e9eebd953a"><code>3a3144f</code></a>
Fix typo: <code>pyproject.license</code> -&gt;
<code>project.license</code> (<a
href="https://redirect.github.com/pypa/setuptools/issues/4931">#4931</a>)</li>
<li><a
href="https://github.com/pypa/setuptools/commit/d751068fd2627d6d8f1729e39cbcd8119049998f"><code>d751068</code></a>
Fix typo: pyproject.license -&gt; project.license</li>
<li>Additional commits viewable in <a
href="https://github.com/pypa/setuptools/compare/v66.1.1...v78.1.1">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=setuptools&package-manager=uv&previous-version=66.1.1&new-version=78.1.1)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)
You can disable automated security fix PRs for this repo from the
[Security Alerts
page](https://github.com/datacommonsorg/website/network/alerts).

</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR automatically updates the `mixer` and `import` submodules to
their latest `master` branches.

Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
…3 usage before going into production (datacommonsorg#6175)

The staging cut last week already includes the gemini switch logic and
this PR is mean to monitor Gemini 3 usage and performance before going
into prod
## Description

The feature flag criteria for determining whether or not VAI was on was
incorrectly named (named using the constant naming convention rather
than the feature flag naming convention).

In environments where V2 was on and VAI was on (but NL server was off)
the Stat Var search was incorrectly being hidden.
This PR automatically updates the `mixer` and `import` submodules to
their latest `master` branches.

Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
…through v2 resolve (datacommonsorg#6179)

This change add a flag to use v2 resolve for the nl_search_var and then
detect-and-fulfill endpoint
…#6161)

## Description

This PR adds the `use_v2_api` flag to production, initially set to
false.
Also deprecate ADMIN_SECRET, since /admin page no longer exists.
This PR automatically updates the `mixer` and `import` submodules to
their latest `master` branches.

Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
This PR automatically updates the `mixer` and `import` submodules to
their latest `master` branches.

Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
@gemini-code-assist
Copy link
Copy Markdown
Contributor

Summary of Changes

Hello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request prepares the Custom Data Commons for a stable release by integrating V2 API features, enhancing UI components with pagination, and performing necessary dependency updates. It also streamlines the codebase by removing deprecated visualization tools and updating feature flag configurations to enable new functionality in production and staging environments.

Highlights

  • Feature Flag Updates: Enabled 'enable_gemini_3_flash' and 'use_v2_api' across multiple environments to support new NL detection and V2 REST API features.
  • Ranking Tile Pagination: Added pagination support to the ranking tile component, allowing users to load more entries dynamically.
  • V2 API Integration: Introduced V2 Stat Var search logic and fallback mechanisms for custom Data Commons, including new API endpoints and ancestor resolution.
  • Dependency Upgrades: Upgraded several core dependencies including google-adk, fastapi, pydantic, and google-genai to their latest stable versions.
  • Code Cleanup: Removed legacy visualization tool components and associated webdriver tests as part of the stable release cleanup.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@gmechali gmechali requested a review from dwnoble April 13, 2026 22:06
Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refactors the Data Commons tools by introducing V2 stat var search logic, turning down the old visualization tool in favor of standardized versions, and adding pagination support to ranking tiles. Backend updates include dependency upgrades and a new V2-based implementation for retrieving variable ancestors. Feedback highlights high-severity issues, including an undefined logger variable in the Python code and missing dependencies in a React useEffect hook that could prevent data from refreshing. Additionally, the new table footer implementation in the ranking units requires structural corrections to avoid breaking the layout.

break

if selected_parent in visited:
logger.error(f"Cycle detected in StatVar hierarchy at {selected_parent}")
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The variable logger is not defined in this scope. Based on the imports at the top of the file (from server.lib import log), you should use log.error instead to log the detected cycle.

Suggested change
logger.error(f"Cycle detected in StatVar hierarchy at {selected_parent}")
log.error(f"Cycle detected in StatVar hierarchy at {selected_parent}")

Comment on lines +179 to 180
props.facetSelector,
]);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The useEffect hook is missing parentPlace and enclosedPlaceType in its dependency array. These variables are used inside the effect to determine if a new data fetch is required (via currentFetchProps). Without them in the dependency array, the tile will not refresh when the user selects a different place or place type.

    parentPlace,
    enclosedPlaceType,
    props.facetSelector,
  ]);

Comment on lines +312 to +313
<td>{props.tableFooter}</td>
<td colSpan={footerColSpan}></td>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The footer row structure is incorrect. footerColSpan is calculated as the total number of columns in the table. By placing the footer content in one td and then adding another td with colSpan={footerColSpan}, the row ends up having 1 + footerColSpan columns, which exceeds the table's column count and will break the layout. It should be a single td spanning all columns.

Suggested change
<td>{props.tableFooter}</td>
<td colSpan={footerColSpan}></td>
<td colSpan={footerColSpan}>{props.tableFooter}</td>

Comment on lines +358 to +359
<td>{props.tableFooter}</td>
<td colSpan={footerColSpan}></td>
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The footer row structure is incorrect here as well. It should use a single td with colSpan={footerColSpan} to correctly span the entire table width.

Suggested change
<td>{props.tableFooter}</td>
<td colSpan={footerColSpan}></td>
<td colSpan={footerColSpan}>{props.tableFooter}</td>

@gmechali gmechali added this pull request to the merge queue Apr 13, 2026
Merged via the queue into datacommonsorg:customdc_stable with commit f9111a1 Apr 13, 2026
29 of 33 checks passed
@gmechali gmechali deleted the cdcStableUpdate branch April 13, 2026 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

9 participants