refactor: set crawl delay rule in robots.txt

slsfi · Mar 1, 2024 · 0413ae6 · 0413ae6
1 parent 54953b9
commit 0413ae6
Show file tree

Hide file tree

Showing 2 changed files with 10 additions and 5 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -8,13 +8,9 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
 
 ## Unreleased
 
-
-
-## [1.3.1-production.1] – 2024-02-19
-
 ### Changed
 
-- Merge release [`v1.3.1`](https://github.com/slsfi/digital-edition-frontend-ng/releases/tag/1.3.1) from upstream, original repository into `production` branch. ([fe500d7](https://github.com/slsfi/topelius-frontend/commit/fe500d7d4048d60f7c8d5beadef720251ad186b3))
+- Set crawl delay rule in robots.txt.
 
 
 
@@ -36,6 +32,14 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/), and this
 
 
 
+## [1.3.1-production.1] – 2024-02-19
+
+### Changed
+
+- Merge release [`v1.3.1`](https://github.com/slsfi/digital-edition-frontend-ng/releases/tag/1.3.1) from upstream, original repository into `production` branch. ([fe500d7](https://github.com/slsfi/topelius-frontend/commit/fe500d7d4048d60f7c8d5beadef720251ad186b3))
+
+
+
 ## [1.3.1] – 2024-02-19
 
 ### Fixes

diff --git a/src/robots.txt b/src/robots.txt
@@ -1,4 +1,5 @@
 User-agent: *
 Disallow: /search
+Crawl-delay: 10
 
 Sitemap: https://topelius.sls.fi/sitemap.txt