[MDS] Install Vega sample data #6218

huyaboo · 2024-03-19T23:50:33Z

Description

With #6123 and #5975 , MDS now fully supports Vega and importing. This PR will fix Vega visualizations when installing sample data. Specifically, the data_source_name field will be appended to each url body of the Vega spec and the datasource reference added to the saved object metadata.

Issues Resolved

Closes #5927

Screenshot

Screen.Recording.2024-03-19.at.4.47.35.PM.mov

Testing the changes

Start with no sample data at all
Add sample data from Data Source Sample
Check Vega Visualizations to see that data_source_name: Data Source Sample was added to each url body of the Vega spec and the datasource reference is included in the saved object metadata.

Check List

All tests pass
- yarn test:jest
- yarn test:jest_integration
New functionality includes testing.
New functionality has been documented.
Update CHANGELOG.md
Commits are signed per the DCO using --signoff

codecov · 2024-03-20T00:04:47Z

Codecov Report

Attention: Patch coverage is 90.00000% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 67.46%. Comparing base (40da92c) to head (3ac41a0).

Files	Patch %	Lines
...home/server/services/sample_data/data_sets/util.ts	87.50%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #6218   +/-   ##
=======================================
  Coverage   67.46%   67.46%           
=======================================
  Files        3368     3368           
  Lines       65422    65431    +9     
  Branches    10557    10560    +3     
=======================================
+ Hits        44134    44145   +11     
+ Misses      18717    18714    -3     
- Partials     2571     2572    +1

Flag	Coverage Δ
Linux_1	`32.14% <0.00%> (-0.01%)`	⬇️
Linux_2	`55.58% <100.00%> (+<0.01%)`	⬆️
Linux_3	`44.90% <0.00%> (-0.02%)`	⬇️
Linux_4	`35.07% <80.00%> (+0.09%)`	⬆️
Windows_1	`32.16% <0.00%> (-0.01%)`	⬇️
Windows_2	`55.55% <100.00%> (+<0.01%)`	⬆️
Windows_3	`44.92% <0.00%> (-0.01%)`	⬇️
Windows_4	`35.07% <80.00%> (+0.09%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

yujin-emma

LGTM

tianleh · 2024-03-21T23:55:12Z

src/core/server/saved_objects/import/utils.ts

 }

 export const updateDataSourceNameInVegaSpec = (
  props: UpdateDataSourceNameInVegaSpecProps
 ): string => {
-  const { spec } = props;
+  const { spec, spacing } = props;
+  const stringifiedSpacing = !!spacing ? spacing : 2;


a shorter way to write this const stringifiedSpacing = spacing || 2;

tianleh · 2024-03-22T00:16:41Z

src/plugins/home/server/services/sample_data/data_sets/util.test.ts

+import { SavedObject, updateDataSourceNameInVegaSpec } from '../../../../../../core/server';
+
+describe('getSavedObjectsWithDataSource()', () => {
+  const getVisualizationSavedObjects = (): Array<SavedObject<any>> => {


nonblocking: you can move this large json object to a data.json file and then load it with one line

import data from './data.json'

seraphjiang · 2024-03-22T01:20:35Z

CHANGELOG.md

@@ -59,6 +59,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Discover] Options button to configure legacy mode and remove the top navigation option ([#6170](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6170))
 - [Multiple Datasource] Add default functionality for customer to choose default datasource ([#6058](https://github.com/opensearch-project/OpenSearch-Dashboards/issues/6058))
 - [Multiple Datasource] Add import support for Vega when specifying a datasource ([#6123](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6123))
+- [Workspace] Validate if workspace exists when setup inside a workspace ([#6154](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6154))


@huyaboo would you rebase from main, so others change won't showup in your PR

The CHANGELOG.md change was because my commit adds a new line at the end of the file

BionIT · 2024-03-26T03:35:11Z

CHANGELOG.md

Did we change this line?

I believe it was a newline added to the end of my CHANGELOG.md

@huyaboo let's rebase with main. Other contributor's change should not show in your PR.

BionIT · 2024-03-26T03:38:16Z

src/core/server/saved_objects/import/utils.ts

 }

 export const updateDataSourceNameInVegaSpec = (
  props: UpdateDataSourceNameInVegaSpecProps
 ): string => {
-  const { spec } = props;
+  const { spec, spacing } = props;
+  const stringifiedSpacing = spacing || 2;


So the type of spacing is number, why is it called stringifiedSpacing? Also, why it is || 2 but not other number? what if spacing is 0?

stringifiedSpacing is how much indenting should occur when an HJSON document is stringified. It's set to 2 as that seems to be the default for most Vega visualizations. If it were 0 there would be no indenting between the lines so it would look something like this:

{ data: { url: { index: some_index body: { ... } } } ... }

Can we add those explanation as comments?

BionIT · 2024-03-26T03:39:44Z

src/plugins/home/server/services/sample_data/data_sets/util.ts

+            const updatedVegaSpec = updateDataSourceNameInVegaSpec({
+              spec: vegaSpec,
+              newDataSourceName: dataSourceTitle,
+              spacing: 1,


why do we hard code spacing as 1 and what does it do?

The reason spacing is 1 is because there is one sample data visualization (sankey visualization) that would cause a URL too long error. This setting limits the spacing so the visualization will still be rendered/accessed.

Can we add those explanation as comments?

xinruiba · 2024-03-28T23:55:47Z

LGTM, thanks

bandinib-amzn · 2024-03-28T23:54:42Z

CHANGELOG.md

@@ -71,7 +71,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 - [Workspace] Add create workspace page ([#6179](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6179))
 - [Multiple Datasource] Make sure customer always have a default datasource ([#6237](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6237))
 - [Workspace] Add workspace list page ([#6182](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6182))
-
+- [Multiple Datasource] Append `data_source_name` to the Vega spec and add datasource reference when installing sample data ([#6218](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/6218))


nit: can we add simpler text instead of adding details so that it would be easy to understand by anyone? maybe [Multiple Datasource] Add multi data source support to sample vega visualizations

bandinib-amzn · 2024-03-28T23:56:07Z

src/core/server/saved_objects/import/utils.ts

@@ -6,15 +6,24 @@
 import { parse, stringify } from 'hjson';
 import { SavedObject, SavedObjectsClientContract } from '../types';

+/**
+ * Given a Vega spec, the new datasource (by name), and spacing,


It's not clear from this line, what exactly this function is doing.

bandinib-amzn · 2024-03-29T00:00:53Z

Can you update with rebase before merge?

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

opensearch-trigger-bot · 2024-03-29T04:58:23Z

The backport to 2.x failed:

The process '/usr/bin/git' failed with exit code 128

To backport manually, run these commands in your terminal:

# Navigate to the root of your repository
cd $(git rev-parse --show-toplevel)
# Fetch latest updates from GitHub
git fetch
# Create a new working tree
git worktree add ../.worktrees/OpenSearch-Dashboards/backport-2.x 2.x
# Navigate to the new working tree
pushd ../.worktrees/OpenSearch-Dashboards/backport-2.x
# Create a new branch
git switch --create backport/backport-6218-to-2.x
# Cherry-pick the merged commit of this pull request and resolve the conflicts
git cherry-pick -x --mainline 1 8810f085164243532bf8d87f3952c30172ff8d22
# Push it to GitHub
git push --set-upstream origin backport/backport-6218-to-2.x
# Go back to the original working tree
popd
# Delete the working tree
git worktree remove ../.worktrees/OpenSearch-Dashboards/backport-2.x

Then, create a pull request where the base branch is 2.x and the compare/head branch is backport/backport-6218-to-2.x.

* Add data_source_name to sample data install Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add unit test coverage Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Resolve CHANGELOG.md conflicts Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Refactor logic Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Remove newline Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Remove newline Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add comments for spacing Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Update jsdoc Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> --------- Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> (cherry picked from commit 8810f08)

* Add data_source_name to sample data install Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add unit test coverage Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Resolve CHANGELOG.md conflicts Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Refactor logic Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Remove newline Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Remove newline Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Add comments for spacing Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> * Update jsdoc Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> --------- Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com> (cherry picked from commit 8810f08) Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

github-actions bot added the first-time-contributor label Mar 19, 2024

huyaboo changed the title ~~Install vega sample data~~ [MDS] Install Vega sample data Mar 20, 2024

huyaboo marked this pull request as ready for review March 20, 2024 18:09

huyaboo requested review from ananzh, kavilla, AMoo-Miki, ashwin-pc, joshuarrrr, abbyhu2000, zengyan-amazon, zhongnansu, manasvinibs, ZilongX, Flyingliuhub, BSFishy, curq, bandinib-amzn, SuZhou-Joe, ruanyl, BionIT and xinruiba as code owners March 20, 2024 18:09

xinruiba added the multiple datasource multiple datasource project label Mar 20, 2024

yujin-emma approved these changes Mar 21, 2024

View reviewed changes

ZilongX added backport 2.x v2.14.0 labels Mar 21, 2024

tianleh reviewed Mar 21, 2024

View reviewed changes

tianleh reviewed Mar 22, 2024

View reviewed changes

seraphjiang reviewed Mar 22, 2024

View reviewed changes

huyaboo force-pushed the install-vega-sample-data branch from f006109 to 94fcc10 Compare March 22, 2024 21:24

tianleh approved these changes Mar 22, 2024

View reviewed changes

BionIT reviewed Mar 26, 2024

View reviewed changes

BionIT previously approved these changes Mar 27, 2024

View reviewed changes

huyaboo dismissed BionIT’s stale review via 6d54eaf March 27, 2024 20:08

BionIT previously approved these changes Mar 28, 2024

View reviewed changes

xinruiba previously approved these changes Mar 28, 2024

View reviewed changes

bandinib-amzn reviewed Mar 28, 2024

View reviewed changes

huyaboo added 7 commits March 29, 2024 00:21

Add data_source_name to sample data install

854a32e

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Add unit test coverage

001d499

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Resolve CHANGELOG.md conflicts

32b0575

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Refactor logic

09dbbdf

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Remove newline

9435b6c

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Remove newline

3dcf3a0

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

Add comments for spacing

c000459

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

huyaboo dismissed stale reviews from xinruiba and BionIT via c000459 March 29, 2024 00:28

huyaboo force-pushed the install-vega-sample-data branch from bca768d to c000459 Compare March 29, 2024 00:28

Update jsdoc

3ac41a0

Signed-off-by: Huy Nguyen <73027756+huyaboo@users.noreply.github.com>

bandinib-amzn approved these changes Mar 29, 2024

View reviewed changes

BionIT approved these changes Mar 29, 2024

View reviewed changes

BionIT merged commit 8810f08 into opensearch-project:main Mar 29, 2024
68 checks passed

opensearch-trigger-bot bot added the failed backport label Mar 29, 2024

huyaboo deleted the install-vega-sample-data branch March 29, 2024 16:34

huyaboo mentioned this pull request Apr 1, 2024

[manual backport 2.x][MDS] Install Vega sample data (#6218) #6299

Merged

huyaboo removed the failed backport label Apr 24, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[MDS] Install Vega sample data #6218

[MDS] Install Vega sample data #6218

huyaboo commented Mar 19, 2024 •

edited

codecov bot commented Mar 20, 2024 •

edited

yujin-emma left a comment

tianleh Mar 21, 2024

tianleh Mar 22, 2024

seraphjiang Mar 22, 2024

huyaboo Mar 22, 2024

BionIT Mar 26, 2024

huyaboo Mar 26, 2024

bandinib-amzn Mar 29, 2024

BionIT Mar 26, 2024

huyaboo Mar 26, 2024

BionIT Mar 27, 2024

BionIT Mar 26, 2024

huyaboo Mar 26, 2024

BionIT Mar 27, 2024

xinruiba commented Mar 28, 2024

bandinib-amzn Mar 28, 2024

bandinib-amzn Mar 28, 2024

bandinib-amzn commented Mar 29, 2024

opensearch-trigger-bot bot commented Mar 29, 2024

[MDS] Install Vega sample data #6218

[MDS] Install Vega sample data #6218

Conversation

huyaboo commented Mar 19, 2024 • edited

Description

Issues Resolved

Screenshot

Testing the changes

Check List

codecov bot commented Mar 20, 2024 • edited

Codecov Report

yujin-emma left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xinruiba commented Mar 28, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bandinib-amzn commented Mar 29, 2024

opensearch-trigger-bot bot commented Mar 29, 2024

huyaboo commented Mar 19, 2024 •

edited

codecov bot commented Mar 20, 2024 •

edited