Replace search engine and improve UX #337

davidjgoss · 2023-11-18T16:03:36Z

🤔 What's changed?

elasticlunr is removed in favour of Orama. This gets rid of bundling issues and allows us to do a production build of Ladle and deploy the preview with Netlify. It also reduces the dependency footprint (see lockfile diff).

The diff is bigger than you might expect for simply swapping out a library that does the same thing, because all Orama's interfaces (for creating indexes, and inserting, and searching) are async. This required a refactor of how and when we build the search to fit that async pattern. As such, lot of tests needed converting to be async and having assertions wrapped in await waitFor() etc.

Also, search plumbing is now extracted from the <FilteredResults/> component and done in a hook useFilteredDocuments().

Finally, applied the search query on change (with half-second debounce) so you don't have to hit enter each time - this often frustrates me as a user.

⚡️ What's your motivation

elasticlunr is unmaintained, and also published in a format that is troublesome for some module bundlers e.g. Vite that is behind Ladle. We need to replace it with something more modern, and while we're at it move logic out of components and into hooks/functions where possible.

🏷️ What kind of change is this?

🏦 Refactoring/debt/DX (improvement to code design, tooling, documentation etc. without changing behaviour)

📋 Checklist:

I agree to respect and uphold the Cucumber Community Code of Conduct
I've changed the behaviour of the code
- I have added/updated tests to cover my changes.
My change requires a change to the documentation.
- I have updated the documentation accordingly.
Users should know about my change
- I have added an entry to the "Unreleased" section of the CHANGELOG, linking to this pull request.

This text was originally generated from a template, then edited by hand. You can modify the template here.

netlify · 2023-11-18T16:03:40Z

✅ Deploy Preview for cucumber-react-components ready!

Name	Link
🔨 Latest commit	`0950c24`
🔍 Latest deploy log	https://app.netlify.com/sites/cucumber-react-components/deploys/656c8a3a1e526e00080c3e8e
😎 Deploy Preview	https://deploy-preview-337--cucumber-react-components.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

# Conflicts: # package-lock.json # package.json # src/components/app/FilteredResults.spec.tsx # src/components/app/FilteredResults.tsx # src/filter/filterByStatus.spec.ts # src/search/FeatureSearch.spec.ts # src/search/RuleSearch.spec.ts # src/search/ScenarioSearch.spec.ts # src/search/Search.spec.ts # src/search/Search.ts # src/search/StepSearch.spec.ts # src/search/TagSearch.spec.ts # src/search/TextSearch.spec.ts # src/search/TextSearch.ts

# Conflicts: # .github/workflows/test.yml # package-lock.json # src/components/app/FilteredResults.spec.tsx # src/filter/filterByStatus.spec.ts # src/search/FeatureSearch.spec.ts # src/search/RuleSearch.spec.ts # src/search/ScenarioSearch.spec.ts # src/search/Search.spec.ts # src/search/StepSearch.spec.ts # src/search/TagSearch.spec.ts

davidjgoss · 2023-12-02T10:25:00Z

src/search/ScenarioLikeSearch.ts

+ * Can be used for Backgrounds, Scenarios and Rules, searching against the
+ * name and description
+ */
+class ScenarioLikeSearch<T extends ScenarioLike> implements TypedIndex<T> {


Uses duck typing to avoid the need for specific classes for Scenario, Rule or Background

davidjgoss · 2023-12-02T10:25:17Z

src/filter/filterByStatus.ts

@@ -16,12 +19,12 @@ export default function filterByStatus(
      const pickleIds = gherkinQuery.getPickleIds(gherkinDocument.uri, scenario.id)

      return pickleIds
-        .map((pickleId) =>
+        .filter((pickleId) => envelopesQuery.hasTestCase(pickleId))
+        .some((pickleId) =>


More efficient

davidjgoss · 2023-12-02T10:28:07Z

src/countScenariosByStatuses.ts

@@ -27,28 +37,15 @@ export default function countScenariosByStatuses(
 } {
  const scenarioCountByStatus = makeEmptyScenarioCountsByStatus()

-  for (const gherkinDocument of gherkinQuery.getGherkinDocuments()) {


Since this is totals for the whole test run, it's simpler and faster to interrogate cucumber data directly than go through all the Gherkin documents.

davidjgoss · 2023-12-03T14:02:21Z

vite.config.mjs

Was only needed to work around elasticlunr issues.

davidjgoss · 2023-12-03T14:11:09Z

src/components/app/FilteredResults.tsx

        />
      </div>

-      {filtered.length > 0 && <GherkinDocumentList gherkinDocuments={filtered} preExpand={true} />}
-      {filtered.length < 1 && <NoMatchResult query={query} />}
+      {filtered !== undefined && (


filtered can be undefined if the (async) search initialisation hasn't happened yet - in which case we just won't render anything yet in this area.

davidjgoss · 2023-12-03T14:11:28Z

src/components/app/FilteredResults.tsx

@@ -52,15 +36,22 @@ export const FilteredResults: React.FunctionComponent<IProps> = ({ className })
        />
        <SearchBar
          query={query}
-          onSearch={(query) => update({ query })}
+          onSearch={(newValue) => update({ query: newValue })}


This was shadowing a variable higher in the scope.

davidjgoss · 2023-12-03T14:11:34Z

src/components/app/FilteredResults.tsx

          statusesWithScenarios={statusesWithScenarios}
          hideStatuses={hideStatuses}
-          onFilter={(hideStatuses) => update({ hideStatuses })}
+          onFilter={(newValue) => update({ hideStatuses: newValue })}


davidjgoss · 2023-12-03T14:13:15Z

src/components/app/SearchBar.tsx

-    const formData = new window.FormData(event.currentTarget)
-    const query = formData.get('query')
-    onSearch((query || '').toString())
+    debouncedSearchChange.flush()


flush() == "fast forward and apply any pending changes now"

src/components/app/SearchBar.spec.tsx

Recent changes to `@cucumber/react-components` [1] makes rendering async. This is a good opportunity to switch to a library providing query polling. [1] cucumber/react-components#337

davidjgoss added 2 commits November 18, 2023 15:39

stop shadowing variables

bf3d06d

simplify function

e58cf14

davidjgoss added 2 commits November 18, 2023 16:49

split search and filtering code to a hook

e45caf5

more hookifying

98ba085

davidjgoss mentioned this pull request Nov 18, 2023

Add Netlify deployment of Ladle #12

Closed

davidjgoss added 9 commits November 21, 2023 20:43

split out a createSearch function

c9fcc62

move searchable and results into state

d5c4de7

make creation interface async

d9978ab

make search method async

61b645b

make more of the search async, use interfaces

3baf6de

reimplement step search with orama

771f884

reimplement rule search with orama

72f4b6d

reimplement feature search with orama

2ffa44b

reimplement background and scenario searches, consolidate

6d6cd96

davidjgoss mentioned this pull request Nov 29, 2023

Switch to ESM #338

Merged

7 tasks

davidjgoss added 5 commits November 30, 2023 16:17

fix some more compilation issues

5de8b75

some cleaning up

de2f245

chore: use Node.js 18 and 20 in CI (#339)

a2dea4a

fix lockfile issues maybe

f530426

davidjgoss force-pushed the feat/change-search-engine branch from 2d57890 to f530426 Compare November 30, 2023 17:15

davidjgoss mentioned this pull request Dec 2, 2023

switch to mocha for testing #340

Merged

7 tasks

davidjgoss added 2 commits December 2, 2023 10:11

use orama stemmer, remove elasticlunr

aeec0cd

davidjgoss commented Dec 2, 2023

View reviewed changes

davidjgoss added 2 commits December 2, 2023 10:31

rework types a bit

1986844

clarify interfaces and types

c694f37

implement debounced search change handling

b98596d

davidjgoss force-pushed the feat/change-search-engine branch from c0d0554 to b98596d Compare December 2, 2023 21:28

davidjgoss changed the title ~~replace search engine~~ Replace search engine and improve UX Dec 2, 2023

update CHANGELOG.md

5df2785

davidjgoss marked this pull request as ready for review December 3, 2023 13:55

davidjgoss added 2 commits December 3, 2023 13:57

chore: update and re-enable test

aeab13e

chore: update and re-enable test

0950c24

davidjgoss commented Dec 3, 2023

View reviewed changes

vite.config.mjs

Copy link

Contributor Author

davidjgoss Dec 3, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was only needed to work around elasticlunr issues.

davidjgoss requested review from aslakhellesoy and olleolleolle December 3, 2023 14:03

davidjgoss commented Dec 3, 2023

View reviewed changes

mpkorstanje reviewed Dec 4, 2023

View reviewed changes

src/components/app/SearchBar.spec.tsx Show resolved Hide resolved

mpkorstanje approved these changes Dec 8, 2023

View reviewed changes

davidjgoss merged commit 405c88d into main Dec 9, 2023
6 checks passed

davidjgoss deleted the feat/change-search-engine branch December 9, 2023 12:17

davidjgoss mentioned this pull request Dec 9, 2023

New html formatter produces very large report files cucumber/html-formatter#62

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace search engine and improve UX #337

Replace search engine and improve UX #337

davidjgoss commented Nov 18, 2023 •

edited

netlify bot commented Nov 18, 2023 •

edited

davidjgoss Dec 2, 2023

davidjgoss Dec 2, 2023

davidjgoss Dec 2, 2023

davidjgoss Dec 3, 2023

davidjgoss Dec 3, 2023

davidjgoss Dec 3, 2023

davidjgoss Dec 3, 2023

davidjgoss Dec 3, 2023

Replace search engine and improve UX #337

Replace search engine and improve UX #337

Conversation

davidjgoss commented Nov 18, 2023 • edited

🤔 What's changed?

⚡️ What's your motivation

🏷️ What kind of change is this?

📋 Checklist:

netlify bot commented Nov 18, 2023 • edited

✅ Deploy Preview for cucumber-react-components ready!

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

davidjgoss commented Nov 18, 2023 •

edited

netlify bot commented Nov 18, 2023 •

edited