0.9.0

Added

• Catalan and Romanian sites (locales ca, ro_RO) are analyzed with the
  catalan and romanian analyzers instead of the standard one.
• When a version of the plugin changes the built-in analyzer of a language
  of the site (option tokenizer_engine_for_wp_language_analyzers), the
  index is rebuilt in the background: once after this update on a Catalan or
  Romanian site, on the next ones for the languages they add.
• Export of the settings of Elasticsearch or OpenSearch indices that analyze
  like the plugin, to move to a search server:
  wp tokenizer-engine-for-wp index export [<index>] (one index per post
  type, taxonomy and language; analyzers, fields and their weights). Site
  Health points to it beyond about 10,000 contents.

Changed

• Requires lalbert/tokenizer-engine 1.3 (Catalan and Romanian analyzers;
  bare domain names in uax_url_email; standard with max_token_length
  reads the rest of a longer word as a new word, like Elasticsearch).
• After a change of the index, related content is recomputed only for the
  sources it affects: the changed objects, the sources that list them and
  the 50 closest to them (option tokenizer_engine_for_wp_related_changes).
  One post updated: 57 sources recomputed in 2.4 s instead of 2 min on
  5,000 posts, 54 sources in 11 s instead of 32 min on 20,000 posts. Every
  source is still recomputed every night, after a rebuild and beyond 200
  grouped changes. benchmarks/scale.sh measures it.

Fixed

• Sites in Dutch, Swedish, Norwegian, Danish, Finnish or Russian failed to
  index (error "Snowball stemmers require the wamania/php-stemmer package"):
  the plugin now requires wamania/php-stemmer (MIT), which their analyzers
  and the Catalan and Romanian ones need.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.9.0/CHANGELOG.md

0.8.0

Added

• Search statistics: with the setting "Searches of visitors" (formerly
  "Searches without results"), every search of a visitor is counted per
  day, with its number of results. Screen Tools > Searches: the searches
  of the last 7, 30 or 90 days, most frequent first, the number of
  searches and the share without results, views Frequent and Without
  results, delete a search or clear the list. It replaces the list at the
  bottom of the settings. Schema version 7 (table search_days, the
  searches without results kept so far are moved into it).
• Site Health (Tools > Site Health): tests of the index (something to
  index, built), of the background tasks (WP-Cron late) and of the size of
  the index (up to about 10,000 contents), and a section of the Info tab
  with the state of the index and of the related content. The last
  complete recomputation of the related content is recorded (option
  tokenizer_engine_for_wp_related_last).
• Scale benchmark benchmarks/scale.sh (N synthetic posts: indexing,
  related content, response times).

Changed

• The recomputation of related content reads the statistics of the index
  (document frequencies, lengths) once per batch instead of once per
  content, in batches of 100: 12 times faster on 5,000 posts (24 min to
  2 min).
• The link suggestions of the editor only analyze the titles whose every
  word is in the text, found in the index: 8 times faster on 20,000 posts
  (16 s to 2 s).

Fixed

• A search with invalid UTF-8 in the address (?s=caf%C3, REST parameter
  q) failed with an error 500: invalid bytes are now replaced, in the
  search, the highlighting and the other APIs taking a text.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.8.0/CHANGELOG.md

0.7.0

Added

• Pages not found (404): the contents closest to the address requested
  (its words searched in the index: stemming, synonyms, language), without
  any redirect, and the page keeps its 404 status. Inserted at the end of
  the main area of the 404 page (option, off by default: Block Hooks in
  block themes, before </main> in classic themes), block "Suggestions for
  a page not found", shortcode [tokenizer_engine_404], function
  tokenizer_engine_for_wp_not_found(), template
  tokenizer-engine-for-wp/not-found.php, default styles that can be
  turned off. API Engine::suggestForPath() (PathSuggestion: share of the
  words found, clear match), filters not_found_query, not_found_ignore,
  not_found_args, not_found_suggestions. Files and probes of bots are
  ignored.

• Screen Tools > Pages not found (option, off by default): the paths not
  found by visitors, most frequent first, with their suggestions, the
  share of the words found, "Clear match", and a button that copies the
  address of a suggestion, to create the redirect in a redirection plugin.
  Paths can be ignored or deleted. The path only, kept 90 days.

• Contents set to noindex in Yoast SEO, Rank Math, SEOPress, All in One SEO
  or The SEO Framework are left out of the search, related content, link
  targets and suggestions of the 404 page (setting "SEO plugins", on by
  default). They stay indexed, keeping their own related content. The
  noindex is read at indexing: a change of the SEO setting of a content
  reindexes it, a change of the defaults of an SEO plugin rebuilds the
  index. Filter noindex. Schema version 5 (column noindex).

• Internal link counts: the links of each indexed post (by hand, automatic,
  added at display time) are recorded when it is indexed, resolved by
  data-type/data-id or by address (posts, term archives). Column "Links
  in / out" in the lists of the indexed post types, and screen Tools >
  Internal links: orphan contents (no link in, in no menu), contents with 2
  links in or fewer, dead ends, with the related contents an orphan could
  be linked from. Schema version 6 (table links); sites indexed before are
  rebuilt once.

• Typos ("Did you mean"): a search that finds nothing is searched again with
  its misspelled words corrected (closest indexed term, Levenshtein distance
  1 or 2 by length), and the search of the site shows "Showing results for
  X. Search instead for Y". SearchResults::$correction, argument and REST
  parameter correct, REST field correction, setting "Typos" (on by
  default), filters correction and correction_notice. The suggestions of
  the 404 page are corrected too.

Fixed

• The strings of the screen Tools > Automatic links written in
  JavaScript are translated (they were missing from the translation files).

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.7.0/CHANGELOG.md

0.6.0

Added

• Multilingual sites with Polylang or WPML: every content is indexed with
  its language and analyzed with the analyzer of that language; search
  (current language by default, argument and REST parameter language),
  related content, suggested terms and internal links stay within one
  language. Activating or deactivating Polylang or WPML rebuilds the index.
  Filters language_provider, language, current_language for other
  plugins; analysis_settings receives the locale.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.6.0/CHANGELOG.md

0.5.0

Added

• Screen Tools > Automatic links: preview of a post type in batches, then
  automatic links applied or removed post by post or for every post listed
  (REST routes auto-links/preview, auto-links/apply/{id},
  auto-links/remove/{id}).
• Networks of sites (multisite): the tables of a deleted site are dropped,
  and deleting the plugin cleans every site. Tested in CI.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.5.0/CHANGELOG.md

0.4.1

Added

• Links in the list of plugins: "Settings", and the documentation, the
  changelog and the issues of the plugin.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.4.1/CHANGELOG.md

0.4.0

Added

• Synonyms of the search (settings screen, filter synonyms): rules in Solr
  format, applied to the query without rebuilding the index, and
  highlighted in the results.
• Searches without results (option, off by default): counted and listed on
  the settings screen, the query only, kept 90 days.
• Cache of the REST search route: 5 minutes in the object cache and
  Cache-Control for visitors (filter search_cache).

Changed

• The query of the REST search route is limited to 200 characters.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.4.0/CHANGELOG.md

0.3.1

Fixed

• Uninstalling removes every option and transient of the plugin, including
  the generation of the automatic links and their caches
  (tokenizer_engine_for_wp_links_*), the state of related content and the
  shortcode hints.
• Search-as-you-type no longer expands a word longer than 30 characters:
  the SQL query listed every beginning of the typed word.

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.3.1/CHANGELOG.md

0.3.0

Added

• Automatic internal links: Links::apply() and Links::remove(), WP-CLI
  auto-links apply|remove (with --dry-run), or at display time (option).
  Guardrails: maximum per content (3 by default), one link per target, none
  to the content itself or to a target already linked, none in headings
  (option), code or existing links, excluded sources and targets, whole words
  only. Inline formatting crossing a link is closed and reopened around it.
  Automatic links carry data-te-link="auto", data-type and data-id;
  filters auto_link, auto_link_attributes, auto_link_skipped_blocks.
• The documentation and its French translation change together (CI check).

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.3.0/CHANGELOG.md

0.2.0

Added

• French documentation: README.fr.md and docs/fr/api.md (the English
  documentation remains the reference).
• Internal link suggestions in the block editor (panel "Suggestions"): the
  link is inserted in the block containing the anchor.
• Suggested terms for a post or a text (Engine::suggestTerms(), REST
  POST /terms/suggest), with an "Add" button in the block editor.
• Suggestions while typing in the search fields of the site (option), and
  search-as-you-type in the API (prefix).
• Related content of terms on their edit screen (pinned, excluded).
• Preview of the text indexed with each shortcode policy.
• REST: titles of related items and link targets, exclude for link
  suggestions.
• Browser tests (Playwright) and WP-CLI tests in CI.

Changed

• The editor panel lists related content of every enabled post type and
  taxonomy, not only the type of the post.
• Link anchors no longer start with an elision ("observer les baleines", not
  "d’observer les baleines").

Fixed

• The block "Related content" was not supported in the editor (its script
  had no declared dependencies).
• The editor panel showed IDs instead of titles.
• Highlighting of search results was lost in block themes.
• Changing the language of the site now rebuilds the index (the analyzer
  follows it).

Full changelog: https://github.com/lalbert/tokenizer-engine-for-wp/blob/0.2.0/CHANGELOG.md