2026-04-13 Custom DC stable release#6201
2026-04-13 Custom DC stable release#6201gmechali merged 24 commits intodatacommonsorg:customdc_stablefrom
Conversation
This PR removes the codepaths that supported the old, un-revamped /tools/map UI. # Summary Now that the "standardized_vis_tool" experience is the default experience, this PR removes the old /tools/map experience so that we can eventually delete the flag. In order to keep changes concise, I've only removed the old /tools/map. The /tools/scatter, /tools/timeline, and /tools/visualization will come in a separate PR. # Testing Strategy Locally, I set the "standardized_vis_tool" flag to False, and verified that: * navigating to localhost:8080/tools/map still renders the new, standardized experience. * webdriver, client, and python tests still pass
This PR removes the codepaths that supported the old, un-revamped /tools/timeline UI. # Summary Now that the "standardized_vis_tool" experience is the default experience, this PR removes the old /tools/timeline experience so that we can eventually delete the flag. In order to keep changes concise, I've only removed the old /tools/timeline in this PR. Removing /tools/visualization will come in a separate PR. Related PRs: /tools/map is being removed in datacommonsorg#6165 /tools/scatter is being removed in datacommonsorg#6167 # Testing Strategy Locally, I set the "standardized_vis_tool" flag to False, and verified that: * navigating to localhost:8080/tools/timeline still renders the new, standardized experience. * webdriver, client, and python tests still pass
## Description The webdriver tests recordings were out-of-date and notably did not contain recordings for the metadata/facet endpoints (which are heavy on mixer). This PR * Updates the tests for data drift * Adds hashing and fallback entries for the metadata/facet endpoint * Regenerates the recordings. ## Notes The custom DC tests in some cases currently expect slightly different data (or forms of the same data) than the primary tests. Some tests were updated to make them more flexible.
This PR removes the codepaths that supported the old, un-revamped /tools/scatter UI. # Summary Now that the "standardized_vis_tool" experience is the default experience, this PR removes the old /tools/scatter experience so that we can eventually delete the flag. In order to keep changes concise, I've only removed the old /tools/scatter in this PR. Removing /tools/timeline, and /tools/visualization will come in a separate PR. Related PR: /tools/map is being removed in datacommonsorg#6165 # Testing Strategy Locally, I set the "standardized_vis_tool" flag to False, and verified that: * navigating to localhost:8080/tools/scatter still renders the new, standardized experience. * webdriver, client, and python tests still pass --------- Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
## Issue [b/493339134](https://b.corp.google.com/issues/493339134) ## Description This PR converts the usage of ` /v1/variable/search` to `v2` endpoints centered around `/v2/resolve`. ## Notes In order to keep search functionality for custom DCs while migrating away from the `v1`-centric search performed before, we have moved the logic of the search to use `v2/resolve`. The `search_statvar` endpoint now follows the following logic: * If Vertex AI is active (governed by a flag), we use Vertex AI (this functionality has not changed in this PR) * Else, if the NL server is running and we are a custom DC, we use the new `v2/resolve`-centric method to do the search * Else, we do not allow search (noting that this should not occur in practice, given the above paths). When we are in the `v2/resolve` branch, we follow the following logic: * Resolve Query to Candidate DCIDS: We call `/v2/resolve` to convert the natural language query into a list of candidate DCIDs. * Check Observations: If the request filters by entities, we call `/v2/observation` for all candidates. We discards any variables that don't have active observation data for those entities (we skip this if no entities given) * Filter by typeOf: We calls `/v2/node` with property: "->typeOf" to ensure candidates are actually StatisticalVariables and filters out any StatVarGroups or Topics. * Fetch Names: For the remaining valid DCIDs, we call dc.v2node with property: "->name" to retrieve their human-readable display names. * Format Response: Finally, we merges the DCIDs and display names into a standard result structure ({"statVars": [...], "matches": [], "statVarGroups": []}) to maintain compatibility with existing frontend usage. ## Testing [Stat Var Explorer](http://localhost:8080/tools/statvar) * When running the server normally: `./run_server.sh -m`, a search will use Vertex AI and should be identical to what it was before. * When running the server as a custom DC: `./run_server.sh -m -e custom`, the new search path is invoked. The search experience should generally be the same as in master, but the results may not (because we are using the embeddings-based based `v2/resolve` search that will not surface all variables. * When running the server as a custom DC without the NL server: `./run_server.sh -m -e custom`, not search box should appear in the front end. [Timeline Explorer](http://localhost:8080/tools/timeline) All below are for running as a custom DC `./run_server.sh -m -e custom` * When no place is selected, a search for population will bring up many stat vars (a set of results unfiltered by entity) * When a place is selected (such as India), the same search should produce noticeably fewer results (only results that are relevant for India).
This PR removes /tools/visualization, leaving only the redirect to the new standardized /tools/map, /tools/scatter, /tools/timeline experience. ## Description This PR cleans up the legacy `/tools/visualization` route and its associated UI components. It converts the page into a lightweight client-side redirector that sends users to the standardized `/tools/map`, `/tools/scatter`, or `/tools/timeline` pages based on the URL hash parameters. This is the first part of the cleanup, focusing **only** on the tools/visualization path. Updates to the individual tool pages and their tests will be handled in separate atomic PRs. ## Changes ### 🔧 Server-side Routing & Templates - **`server/routes/tools/html.py`** - Simplified the `/tools/visualization` route handler to render the template without server-side data fetching. - **`server/templates/tools/visualization.html`** - Stripped unused styles and Maps API integrations. ### ⚛️ Client-side App & Redirection - **`static/js/apps/visualization/main.ts`** - Updated to perform immediate client-side redirection based on the URL hash. - **[DELETE] `static/js/apps/visualization/app.tsx`** and associated UI components (except for redirect utilities). - **[DELETE] `static/css/tools/visualization.scss`** and removed its entry from `webpack.config.js`. ### 🛠️ Shared Utilities - **`static/js/utils/app/visualization_utils.tsx`** - Decoupled from the deleted `app_context.ts` by defining `ContextStatVar` and `DisplayOptions` interfaces locally. This unblocks chart tiles that still share these utilities without pulling in old visualization app context. ## Testing Strategy ### Manual Verification 1. Navigate to `/tools/visualization#visType=timeline` (or map/scatter) and verify that it immediately redirects to the new `/tools/timeline` page (or map/scatter) with the hash parameters intact. 2. Verify that the build completes successfully without `app_context` missing errors. 3. Verify that even with standardizd_vis_tool flag set to False, redirect still happens as intended 4. Verify that webdriver, python, and client tests pass locally.
This PR automatically updates the `mixer` and `import` submodules to their latest `master` branches. Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
## Description This PR sets `use_v2_api` to true in local, autopush and dev.
…commonsorg#6147) Adds a optional "show next" feature to the ranking tile, allowing users to incrementally load additional entries (e.g. top 100 -> top 200), and adds this functionality to the new ranking pages. # Summary This PR adds an optional button to the ranking tile that when clicked, will expand the ranking tile table to show more entries. The number of entries to add on is configured via the new optional `showNextCount` argument. If showNextCount is 0 or not provided, the button is hidden and pagination is disabled. Features / UI Considerations: * The button is being added in the bottom right corner, where the chart footer is. * If both highest and lowest tables are being shown in the tile, each table gets its own button, and their pagination act independently of each other. * A slide down animation has been added for when the user clicks on the button to make the transition less jarring * This feature has also been enabled on the web components via a new `showNextCount` html attribute * This feature is available on chart configs via the `showNextCount` RankingTileSpec field. * Updated the dev subject page at /dev/subject-page-tiles to include all variants of the ranking tile. Engineering Considerations: * This button is NOT being added to the explore page or topic pages, so there is no expected visual or functional changes to those pages. This button is only being used on the new Ranking pages. * Because the button is an icon only, it does not require i18n strings * Because the ranking metadata used by the subject page configs is intended to follow protobuf conventions, the callback for updating the state of the ranking chart can't be passed in via config. In order to avoid ranking-specific prop drilling through the subject page react components, I've elected to use a ranking page context instead. Note: This PR is the final PR needed for implementation of the new ranking pages. However, in order to prioritize completing the v0/v1 migration as fast as possible, I'll prioritize removing the ranking pages, and come back to add testing for the new ranking page UI and this new button functionality in a followup PR. # Left for future PRs * Updating the web-component documentation on the docsite repo to include the new `showNextCount` attribute. * Adding tests for the new button # Screenshots ### New ranking page UI <img width="3226" height="5846" alt="localhost_8080_ranking_Count_Person_County_country_USA (1)" src="https://github.com/user-attachments/assets/febc6599-b54f-4526-b86f-ca3f8b3d5b86" /> ### Dev page with ranking tiles with and without pagination <img width="2560" height="1328" alt="image" src="https://github.com/user-attachments/assets/a8af7557-c060-4db1-ae41-41dda85633c0" /> --------- Co-authored-by: Nick Blumberg <nblumberg@google.com>
## Description The pinned version of `google-adk` had a security vulnerability. This PR upgrades that library, along with other libraries that needed to be upgrade as part of the dependency cascade. ## Notes This resulted in a number of dependency upgrades (visible in the updated `pyproject.toml` file. I upgraded the minimal number of libraries possible to allow us to update `google-adk`. One of the updates `setuptools` being dropped as the dependency tree. This is explicitly restored at a version that allows us to run our test suites.
Bumps [setuptools](https://github.com/pypa/setuptools) from 66.1.1 to 78.1.1. <details> <summary>Changelog</summary> <p><em>Sourced from <a href="https://github.com/pypa/setuptools/blob/main/NEWS.rst">setuptools's changelog</a>.</em></p> <blockquote> <h1>v78.1.1</h1> <h2>Bugfixes</h2> <ul> <li>More fully sanitized the filename in PackageIndex._download. (<a href="https://redirect.github.com/pypa/setuptools/issues/4946">#4946</a>)</li> </ul> <h1>v78.1.0</h1> <h2>Features</h2> <ul> <li>Restore access to _get_vc_env with a warning. (<a href="https://redirect.github.com/pypa/setuptools/issues/4874">#4874</a>)</li> </ul> <h1>v78.0.2</h1> <h2>Bugfixes</h2> <ul> <li>Postponed removals of deprecated dash-separated and uppercase fields in <code>setup.cfg</code>. All packages with deprecated configurations are advised to move before 2026. (<a href="https://redirect.github.com/pypa/setuptools/issues/4911">#4911</a>)</li> </ul> <h1>v78.0.1</h1> <h2>Misc</h2> <ul> <li><a href="https://redirect.github.com/pypa/setuptools/issues/4909">#4909</a></li> </ul> <h1>v78.0.0</h1> <h2>Bugfixes</h2> <ul> <li>Reverted distutils changes that broke the monkey patching of command classes. (<a href="https://redirect.github.com/pypa/setuptools/issues/4902">#4902</a>)</li> </ul> <h2>Deprecations and Removals</h2> <ul> <li>Setuptools no longer accepts options containing uppercase or dash characters in <code>setup.cfg</code>.</li> </ul> <!-- raw HTML omitted --> </blockquote> <p>... (truncated)</p> </details> <details> <summary>Commits</summary> <ul> <li><a href="https://github.com/pypa/setuptools/commit/8e4868a036b7fae3208d16cb4e5fe6d63c3752df"><code>8e4868a</code></a> Bump version: 78.1.0 → 78.1.1</li> <li><a href="https://github.com/pypa/setuptools/commit/100e9a61ad24d5a147ada57357425a8d40626d09"><code>100e9a6</code></a> Merge pull request <a href="https://redirect.github.com/pypa/setuptools/issues/4951">#4951</a></li> <li><a href="https://github.com/pypa/setuptools/commit/8faf1d7e0ca309983252e4f21837b73ee12e960f"><code>8faf1d7</code></a> Add news fragment.</li> <li><a href="https://github.com/pypa/setuptools/commit/2ca4a9fe4758fcd39d771d3d3a5b4840aacebdf7"><code>2ca4a9f</code></a> Rely on re.sub to perform the decision in one expression.</li> <li><a href="https://github.com/pypa/setuptools/commit/e409e8002932f2b86aae7b1abc8f8c2ebf96df2c"><code>e409e80</code></a> Extract _sanitize method for sanitizing the filename.</li> <li><a href="https://github.com/pypa/setuptools/commit/250a6d17978f9f6ac3ac887091f2d32886fbbb0b"><code>250a6d1</code></a> Add a check to ensure the name resolves relative to the tmpdir.</li> <li><a href="https://github.com/pypa/setuptools/commit/d8390feaa99091d1ba9626bec0e4ba7072fc507a"><code>d8390fe</code></a> Extract _resolve_download_filename with test.</li> <li><a href="https://github.com/pypa/setuptools/commit/4e1e89392de5cb405e7844cdc8b20fc2755dbaba"><code>4e1e893</code></a> Merge <a href="https://github.com/jaraco/skeleton">https://github.com/jaraco/skeleton</a></li> <li><a href="https://github.com/pypa/setuptools/commit/3a3144f0d2887fa37c06550f42a101e9eebd953a"><code>3a3144f</code></a> Fix typo: <code>pyproject.license</code> -> <code>project.license</code> (<a href="https://redirect.github.com/pypa/setuptools/issues/4931">#4931</a>)</li> <li><a href="https://github.com/pypa/setuptools/commit/d751068fd2627d6d8f1729e39cbcd8119049998f"><code>d751068</code></a> Fix typo: pyproject.license -> project.license</li> <li>Additional commits viewable in <a href="https://github.com/pypa/setuptools/compare/v66.1.1...v78.1.1">compare view</a></li> </ul> </details> <br /> [](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores) Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting `@dependabot rebase`. [//]: # (dependabot-automerge-start) [//]: # (dependabot-automerge-end) --- <details> <summary>Dependabot commands and options</summary> <br /> You can trigger Dependabot actions by commenting on this PR: - `@dependabot rebase` will rebase this PR - `@dependabot recreate` will recreate this PR, overwriting any edits that have been made to it - `@dependabot show <dependency name> ignore conditions` will show all of the ignore conditions of the specified dependency - `@dependabot ignore this major version` will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this minor version` will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself) - `@dependabot ignore this dependency` will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself) You can disable automated security fix PRs for this repo from the [Security Alerts page](https://github.com/datacommonsorg/website/network/alerts). </details> Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This PR automatically updates the `mixer` and `import` submodules to their latest `master` branches. Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
…3 usage before going into production (datacommonsorg#6175) The staging cut last week already includes the gemini switch logic and this PR is mean to monitor Gemini 3 usage and performance before going into prod
Generated following https://github.com/datacommonsorg/website/tree/master/tools/nl/nodejs_query_differ#update-goldens Diffs seem due to a data refresh
## Description The feature flag criteria for determining whether or not VAI was on was incorrectly named (named using the constant naming convention rather than the feature flag naming convention). In environments where V2 was on and VAI was on (but NL server was off) the Stat Var search was incorrectly being hidden.
This PR automatically updates the `mixer` and `import` submodules to their latest `master` branches. Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
…through v2 resolve (datacommonsorg#6179) This change add a flag to use v2 resolve for the nl_search_var and then detect-and-fulfill endpoint
…d-fulfill endpoint (datacommonsorg#6180)
…#6161) ## Description This PR adds the `use_v2_api` flag to production, initially set to false.
Also deprecate ADMIN_SECRET, since /admin page no longer exists.
This PR automatically updates the `mixer` and `import` submodules to their latest `master` branches. Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
This PR automatically updates the `mixer` and `import` submodules to their latest `master` branches. Co-authored-by: datacommons-robot-author <datacommons-robot-author@users.noreply.github.com>
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request prepares the Custom Data Commons for a stable release by integrating V2 API features, enhancing UI components with pagination, and performing necessary dependency updates. It also streamlines the codebase by removing deprecated visualization tools and updating feature flag configurations to enable new functionality in production and staging environments. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
There was a problem hiding this comment.
Code Review
This pull request refactors the Data Commons tools by introducing V2 stat var search logic, turning down the old visualization tool in favor of standardized versions, and adding pagination support to ranking tiles. Backend updates include dependency upgrades and a new V2-based implementation for retrieving variable ancestors. Feedback highlights high-severity issues, including an undefined logger variable in the Python code and missing dependencies in a React useEffect hook that could prevent data from refreshing. Additionally, the new table footer implementation in the ranking units requires structural corrections to avoid breaking the layout.
| break | ||
|
|
||
| if selected_parent in visited: | ||
| logger.error(f"Cycle detected in StatVar hierarchy at {selected_parent}") |
There was a problem hiding this comment.
The variable logger is not defined in this scope. Based on the imports at the top of the file (from server.lib import log), you should use log.error instead to log the detected cycle.
| logger.error(f"Cycle detected in StatVar hierarchy at {selected_parent}") | |
| log.error(f"Cycle detected in StatVar hierarchy at {selected_parent}") |
| props.facetSelector, | ||
| ]); |
There was a problem hiding this comment.
The useEffect hook is missing parentPlace and enclosedPlaceType in its dependency array. These variables are used inside the effect to determine if a new data fetch is required (via currentFetchProps). Without them in the dependency array, the tile will not refresh when the user selects a different place or place type.
parentPlace,
enclosedPlaceType,
props.facetSelector,
]);
| <td>{props.tableFooter}</td> | ||
| <td colSpan={footerColSpan}></td> |
There was a problem hiding this comment.
The footer row structure is incorrect. footerColSpan is calculated as the total number of columns in the table. By placing the footer content in one td and then adding another td with colSpan={footerColSpan}, the row ends up having 1 + footerColSpan columns, which exceeds the table's column count and will break the layout. It should be a single td spanning all columns.
| <td>{props.tableFooter}</td> | |
| <td colSpan={footerColSpan}></td> | |
| <td colSpan={footerColSpan}>{props.tableFooter}</td> |
| <td>{props.tableFooter}</td> | ||
| <td colSpan={footerColSpan}></td> |
There was a problem hiding this comment.
f9111a1
Highlights
Resources - REMOVE BEFORE MERGING