[SPARK-33166][DOC] Provide Search Function in Spark docs site #30292

beliefer · 2020-11-09T03:37:19Z

What changes were proposed in this pull request?

In the last few releases, our Spark documentation https://spark.apache.org/docs/latest/ becomes richer. It would nice to provide a search function to make our users find contents faster.

DocSearch is entirely free and automated. This PR will use it to provides search function.

The screenshots show below:

Why are the changes needed?

Let the users of Spark documentation could find the needed information effectively.

Does this PR introduce any user-facing change?

Yes

How was this patch tested?

build on my machine and look on brower.

SparkQA · 2020-11-10T15:04:51Z

Test build #130876 has finished for PR 30292 at commit 9c9d639.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-10T15:33:31Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35482/

SparkQA · 2020-11-10T16:01:41Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35482/

beliefer · 2020-11-11T02:17:08Z

@sarutak I cannot reproduce the two problems you mentioned on my Safari browser. It seems the css style is different. Could you pull and build with Jekyll again?

sarutak · 2020-11-11T04:24:55Z

@beliefer I found the first issue isn't in the latest change. The second issue seems that we move to the online Spark document rather than local document.

Before move:

After move:

Is this intended?

beliefer · 2020-11-11T07:29:21Z

@sarutak Because this PR not merged and not publish to the online Spark document.

sarutak · 2020-11-11T07:44:41Z

@sarutak Because this PR not merged and not publish to the online Spark document.

Yes. But even if this is published, the search results consistently refer to the latest release at the time point.
Imagine that this feature is merged to 3.1 and then 3.2 is released in the future, and someone still uses Spark 3.1.
In that case, even though the user uses this feature for 3.1, the search result refer to the document for 3.2 right?

beliefer · 2020-11-11T08:13:58Z

@sarutak Because this PR not merged and not publish to the online Spark document.

Yes. But even if this is published, the search results consistently refer to the latest release at the time point.
Imagine that this feature is merged to 3.1 and then 3.2 is released in the future, and someone still uses Spark 3.1.
In that case, even though the user uses this feature for 3.1, the search result refer to the document for 3.2 right?

If so, we should apply different id from docsearch when we ready release a new spark version. @gatorsmile

sarutak · 2020-11-11T08:17:16Z

If so, we should apply different id from docsearch when we ready release a new spark version. @gatorsmile

Sounds good if we can do it.

gengliangwang · 2020-11-11T16:56:50Z

@beliefer Thanks for the work!
I just tried the PR in my local setup. The element "Documentation" looks wired to me:

It is always with the same URL as the first search result.
I thought the rectangle outside the "Documentation" is an input text box but it is not.

gengliangwang · 2020-11-11T17:00:43Z

I just try removing the style from dosearch.css and it looks fine:

I don't think we can just copy the style from https://docsearch.algolia.com/docs/styling here

gatorsmile · 2020-11-11T19:23:03Z

If so, we should apply different id from docsearch when we ready release a new spark version. @gatorsmile

Yes. This is true. We need to apply different ID for each new version.

SparkQA · 2020-11-12T15:46:53Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35612/

SparkQA · 2020-11-12T16:11:12Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35612/

sarutak · 2020-11-12T17:53:19Z

retest this please.

SparkQA · 2020-11-12T18:11:17Z

Test build #131016 has finished for PR 30292 at commit 7120c5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-11-12T18:42:32Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35622/

SparkQA · 2020-11-12T19:13:39Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35622/

SparkQA · 2020-11-12T20:14:54Z

Test build #131005 has finished for PR 30292 at commit 7120c5b.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

beliefer · 2020-11-13T02:56:26Z

@gengliangwang I resolved the issue you said.

sarutak · 2020-11-13T07:03:09Z

Seems good to me.

gengliangwang

LGTM

gengliangwang · 2020-11-13T08:48:34Z

Thanks, merging to master

beliefer · 2020-11-13T09:59:00Z

@gengliangwang @sarutak @gatorsmile @maropu @HyukjinKwon Thanks for your help!

### What changes were proposed in this pull request? Make the API key of DocSearch configurable and avoid hardcoding in the HTML template ### Why are the changes needed? After #30292, our Spark documentation site supports searching. However, the default API key always points to the latest release doc. We have to set different API keys for different releases. Otherwise, the search results are always based on the latest documentation(https://spark.apache.org/docs/latest/) even when visiting the documentation of previous releases. As per discussion in #30292 (comment), we should make the API key configurable and set different values for different releases. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test Closes #30409 from gengliangwang/apiKey. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

…lts by version ### What changes were proposed in this pull request? In the discussion #30292 (comment), we planned to apply a new API key for each Spark release. However, it turns that DocSearch supports crawling multiple URLs from one website and filtering by fact key: https://docsearch.algolia.com/docs/config-file/#using-regular-expressions Thanks to the help from shortcuts, our Spark doc supports multiple version now: algolia/docsearch-configs#2868 This PR is to add the fact key in the search script and update the instruction in the comment. ### Why are the changes needed? To support filtering Spark documentation search results by the current document version. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Manual test Closes #30469 from gengliangwang/apiKeyFollowUp. Authored-by: Gengliang Wang <gengliang.wang@databricks.com> Signed-off-by: Takeshi Yamamuro <yamamuro@apache.org>

beliefer and others added 29 commits June 19, 2020 10:36

Reuse completeNextStageWithFetchFailure

4a6f903

Merge remote-tracking branch 'upstream/master'

96456e2

Merge remote-tracking branch 'upstream/master'

4314005

Merge remote-tracking branch 'upstream/master'

d6af4a7

Merge remote-tracking branch 'upstream/master'

f69094f

Merge remote-tracking branch 'upstream/master'

b86a42d

Merge branch 'master' of github.com:beliefer/spark

2ac5159

Merge remote-tracking branch 'upstream/master'

9021d6c

Merge branch 'master' of github.com:beliefer/spark

74a2ef4

Merge remote-tracking branch 'upstream/master'

9828158

Merge remote-tracking branch 'upstream/master'

9cd1aaf

Merge remote-tracking branch 'upstream/master'

abfcbb9

Merge remote-tracking branch 'upstream/master'

07c6c81

Merge remote-tracking branch 'upstream/master'

580130b

Merge branch 'master' of github.com:beliefer/spark

3712808

Merge remote-tracking branch 'upstream/master'

6107413

Merge remote-tracking branch 'upstream/master'

4b799b4

Merge remote-tracking branch 'upstream/master'

ee0ecbf

Merge remote-tracking branch 'upstream/master'

596bc61

Merge remote-tracking branch 'upstream/master'

0164e2f

Merge remote-tracking branch 'upstream/master'

90b79fc

Merge remote-tracking branch 'upstream/master'

2cef3a9

Merge remote-tracking branch 'upstream/master'

c26b64f

Merge remote-tracking branch 'upstream/master'

2e02cd2

Merge remote-tracking branch 'upstream/master'

a6d0741

Merge remote-tracking branch 'upstream/master'

82e5b2c

Merge remote-tracking branch 'upstream/master'

70bbf5d

Try to add doc search.

6f6471c

Optmize code

3607e23

github-actions bot added the DOCS label Nov 9, 2020

Optimize css

7120c5b

gengliangwang approved these changes Nov 13, 2020

View reviewed changes

gengliangwang closed this in f80fe21 Nov 13, 2020

gengliangwang mentioned this pull request Nov 18, 2020

[SPARK-33479][DOC] Make the API Key of DocSearch configurable #30409

Closed

gengliangwang mentioned this pull request Nov 23, 2020

[SPARK-33479][DOC][FOLLOWUP] DocSearch: Support filtering search results by version #30469

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-33166][DOC] Provide Search Function in Spark docs site #30292

[SPARK-33166][DOC] Provide Search Function in Spark docs site #30292

beliefer commented Nov 9, 2020 •

edited

Loading

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

beliefer commented Nov 11, 2020

sarutak commented Nov 11, 2020

beliefer commented Nov 11, 2020 •

edited

Loading

sarutak commented Nov 11, 2020 •

edited

Loading

beliefer commented Nov 11, 2020 •

edited

Loading

sarutak commented Nov 11, 2020

gengliangwang commented Nov 11, 2020 •

edited

Loading

gengliangwang commented Nov 11, 2020 •

edited

Loading

gatorsmile commented Nov 11, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

sarutak commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

beliefer commented Nov 13, 2020

sarutak commented Nov 13, 2020

gengliangwang left a comment

gengliangwang commented Nov 13, 2020

beliefer commented Nov 13, 2020

[SPARK-33166][DOC] Provide Search Function in Spark docs site #30292

[SPARK-33166][DOC] Provide Search Function in Spark docs site #30292

Conversation

beliefer commented Nov 9, 2020 • edited Loading

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

SparkQA commented Nov 10, 2020

beliefer commented Nov 11, 2020

sarutak commented Nov 11, 2020

beliefer commented Nov 11, 2020 • edited Loading

sarutak commented Nov 11, 2020 • edited Loading

beliefer commented Nov 11, 2020 • edited Loading

sarutak commented Nov 11, 2020

gengliangwang commented Nov 11, 2020 • edited Loading

gengliangwang commented Nov 11, 2020 • edited Loading

gatorsmile commented Nov 11, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

sarutak commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

SparkQA commented Nov 12, 2020

beliefer commented Nov 13, 2020

sarutak commented Nov 13, 2020

gengliangwang left a comment

Choose a reason for hiding this comment

gengliangwang commented Nov 13, 2020

beliefer commented Nov 13, 2020

beliefer commented Nov 9, 2020 •

edited

Loading

beliefer commented Nov 11, 2020 •

edited

Loading

sarutak commented Nov 11, 2020 •

edited

Loading

beliefer commented Nov 11, 2020 •

edited

Loading

gengliangwang commented Nov 11, 2020 •

edited

Loading

gengliangwang commented Nov 11, 2020 •

edited

Loading