Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add setting to be added to article frontmatter to block web crawlers #736

Merged
merged 1 commit into from
Dec 13, 2023
Merged

add setting to be added to article frontmatter to block web crawlers #736

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions content/archive/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ images: []
weight: 999
toc: true
aliases:
sitemap_exclude: true
robotsdisallow: true
---
This section is intended to be a space for articles that are no longer needed but still have content that we do not want to delete.

Expand Down
4 changes: 3 additions & 1 deletion content/archive/verify-receipts/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,9 @@ images: []
menu:
archive:
parent: "archive"
weight: 10
weight:
sitemap_exclude: true
robotsdisallow: true
---
# This article is deprecated

Expand Down
2 changes: 2 additions & 0 deletions content/contributing/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ date: 2021-05-20T12:03:27+01:00
lastmod: 2021-05-20T12:03:27+01:00
draft: false
images: []
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/contributing/adding-content/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ date: 2021-05-20T18:01:01+01:00
lastmod: 2021-05-20T18:01:01+01:00
draft: false
images: []
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/contributing/adding-content/adding-articles-to-datatrails-docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ menu:
weight: 3
images: []
toc: true
sitemap_exclude: true
robotsdisallow: true
---

{{< note >}}
Expand Down
2 changes: 2 additions & 0 deletions ...contributing/adding-content/adding-content-sections-to-datatrails-docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ menu:
weight: 4
images: []
toc: true
sitemap_exclude: true
robotsdisallow: true
---

{{< note >}}
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/adding-content/creating-your-branch/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ menu:
parent: "adding-content"
weight: 2
toc: true
sitemap_exclude: true
robotsdisallow: true
---

As the main branch of these docs is protected you will need to create a separate branch and submit a PR to make any changes
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/formatting-content/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ date: 2021-05-20T19:14:54+01:00
lastmod: 2021-05-20T19:14:54+01:00
draft: false
images: []
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/contributing/formatting-content/advanced-formatting/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ menu:
parent: "formatting-content"
weight: 7
toc: true
sitemap_exclude: true
robotsdisallow: true
---

As Doks is based on Hugo it has many more extensible formatting options, including some custom tools we have added ourselves to enhance the delivery of documentation overall.
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/formatting-content/basic-markdown-formatting/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ menu:
parent: "formatting-content"
weight: 6
toc: true
sitemap_exclude: true
robotsdisallow: true
---

Doks uses Goldmark markdown which is primarily Github-flavoured.
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/formatting-content/style-guide/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ menu:
parent: "formatting-content"
weight: 8
toc: true
sitemap_exclude: true
robotsdisallow: true
---

It is important in any documentation to have a readable, sane and consistent experience, to that end we have defined some guidelines to ensure all content meets the same standard.
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/getting-started/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,6 @@ date: 2021-05-20T12:03:27+01:00
lastmod: 2021-05-20T12:03:27+01:00
draft: false
images: []
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/contributing/getting-started/pulling-and-building-rkvst-docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ menu:
weight: 1
images: []
toc: true
sitemap_exclude: true
robotsdisallow: true
---
{{< note >}}
DataTrails Docs depend on npm version 14.
Expand Down
2 changes: 2 additions & 0 deletions content/contributing/playground/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,6 @@ lastmod: 2021-05-22T19:47:46+01:00
draft: false
images: []
weight: 999
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/contributing/playground/openapi-example-render/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,8 @@ menu:
parent: "playground"
weight: 999
toc: true
sitemap_exclude: true
robotsdisallow: true
---

{{< openapi url="https://raw.githubusercontent.com/rkvst/archivist-docs/master/doc/openapi/subjectsv1.swagger.json" >}}
2 changes: 2 additions & 0 deletions content/contributing/playground/test-page/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,6 @@ menu:
parent: "playground"
weight: 998
toc: true
sitemap_exclude: true
robotsdisallow: true
---
14 changes: 14 additions & 0 deletions content/sales/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: "Sales"
description: ""
lead: ""
date: 2023-12-11T09:25:37+01:00
lastmod: 2023-12-11T09:25:37+01:00
draft: false
images: []
weight: 999
toc: true
aliases:
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/sales/contactus/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ date: 2021-05-18T18:55:49+01:00
lastmod: 2021-05-18T18:55:49+01:00
draft: false
images: []
sitemap_exclude: true
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did we want to remove sales, support and contact us from crawling?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

offline discussion of archival, vs. deletion of pages no longer used

robotsdisallow: true
---

<a rel="noopener" data-ux-btn="secondary" data-ux="ButtonSecondary" data-aid="CTA_BUTTON_RENDERED" href="http://datatrails.ai/beta" target="_blank" data-tccl="ux2.INTRODUCTION.introduction2.Group.Default.Button.Secondary.47772.click,click" data-typography="ButtonAlpha" class="btn btn-primary">Request a Demo</a>
14 changes: 14 additions & 0 deletions content/support/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
---
title: "Support"
description: ""
lead: ""
date: 2023-12-11T09:25:37+01:00
lastmod: 2023-12-11T09:25:37+01:00
draft: false
images: []
weight: 999
toc: true
aliases:
sitemap_exclude: true
robotsdisallow: true
---
2 changes: 2 additions & 0 deletions content/support/contactus/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ date: 2021-05-18T18:55:36+01:00
lastmod: 2021-05-18T18:55:36+01:00
draft: false
images: []
sitemap_exclude: true
robotsdisallow: true
---

For any queries please contact support@datatrails.ai
5 changes: 3 additions & 2 deletions layouts/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ User-agent: *
{{ if eq (hugo.Environment) "production" -}}
Allow: /
{{ else -}}
Disallow: /
{{ range where .Data.Pages "Params.robotsdisallow" true }}
Disallow: {{ .RelPermalink }}{{ end }}
{{ end }}
Sitemap: {{ "sitemap.xml" | absURL -}}
Sitemap: {{ "sitemap.xml" | absURL -}}
Loading