ZCU-PUB/Add a configurable noindex meta tag for item pages

[milanmajchrak]

Part 1 of dataquest-dev/dspace-customers#851

Adds item.noIndex — a list of item uuids or handles whose item page gets
<meta name="robots" content="noindex, noarchive"> and has its citation_pdf_url dropped.
Non-discoverable items get the tag too. Handle URLs, casing and whitespace are normalized.

• Server-side rendered, so crawlers see the tag in the first HTML response, not after hydration.
• Emitted via addMetaTag(), so clearMetaTags() removes it on the next route change instead of
  leaking the noindex onto every page visited afterwards. A spec asserts the store dispatch.
• No nofollow: crawlers should keep following bitstream links to pick up the X-Robots-Tag
  served for the files themselves.

Ships with an empty list. This config is transferred to the browser unsanitized
(server-init.service.ts saveAppConfigForCSR), so entries would be publicly readable in every
page's HTML — i.e. it would publish which items were suppressed. The two items from the issue are
handled by the nginx counterpart instead, which is needed for the PDFs anyway. Use the list only
where publishing the identifiers is fine.

Guards on Array.isArray() and typeof id === 'string': a scalar noIndex: 123456789/42 passes a
length check and would then throw from the first statement of setDSOMetaTags(), stripping the
title, description and all citation_* tags off every page in the repository.

Tests

metadata.service.spec.ts — 31 SUCCESS (19 existing + 12 new). Reverting the production change
turns 6 of them red, so they are load-bearing. Also green: tsc --noEmit on app/spec/server
tsconfigs, madge --circular, eslint on all touched files, and a 64-spec regression sweep over the
other MetadataService consumers.

Counterpart — required

dataquest-dev/dspace-customers#857. This PR covers landing pages only; a PDF cannot carry a
<meta> tag, so the files need the X-Robots-Tag header.

Deploy

config.yml is baked into the image, so this needs a rebuild and redeploy of customer-zcu-pub —
which will also release #1390, #1401 and #1403. Verify repeatedly after the restart: pm2 runs 9
workers, each with its own bot cache.

curl -s https://dspace.zcu.cz/items/<uuid> | grep -i 'name="robots"'

Not covered here

• Removal requests in Google Search Console / Bing / Seznam — noindex waits for a recrawl.
  GSC is per-property, so naos-be.zcu.cz needs its own or the file URLs cannot be removed.
• Do not add a robots.txt Disallow as a substitute: it blocks the crawl that delivers the
  noindex, freezing existing index entries. Disallow: /handle is already there and worth
  checking in GSC — it prevents already-indexed /handle/ URLs from ever being cleaned.
• Sitemaps still list the items, on purpose — Google must recrawl to see the tag.
• OAI-PMH aggregators keep their own copies. noindex is not access control.

🤖 Generated with Claude Code