A way to not use <!DOCTYPE html> #266

FredKSchott · 2022-08-21T22:09:15Z

FredKSchott
Aug 21, 2022
Maintainer

Note: Migrated from withastro/astro#4386

I'll like a way to stop the doctype from appearing on the page at the top of the response. 😺

https://github.com/reggi/my-astro/blob/main/src/pages/index.astro

Here I'm using an astro file as the endpoint and the component route 2 in one. I'd like to return just json when there's a POST request.

---
import Layout from '../layouts/Layout.astro';
import { Calendar, calendarEndpoint } from '../widgets/Calendar'
let json
if (Astro.request.method === 'POST') {
  const body = await Astro.request.json();
  await calendarEndpoint(body)
  Astro.response.headers.set('Content-Type', 'application/json');
  const json = JSON.stringify({ok: true})
}
---
{ json ? (
  <Fragment set:html={json}></Fragment>
) : (
  <Layout title='hi'>
    <Calendar client:visible/>
  </Layout>
)}

Update (10/17/2022) by @FredKSchott: For the exact use-case above, you can now use Endpoints to return JSON and non-HTML content.

jordemort · 2022-09-13T21:11:00Z

jordemort
Sep 13, 2022

I'd also like this; I want to write an XSLT stylesheet to transform my RSS and Atom feeds into human-friendly versions, and I'd like to reuse my Astro components in the stylesheet.

0 replies

switchtrue · 2022-10-14T01:12:46Z

switchtrue
Oct 14, 2022

I'm using Astro to generate parts of a page that are then used as a Server Side Include into a larger template. Having the appear halfway through the document isn't ideal.

3 replies

FredKSchott Oct 14, 2022
Maintainer Author

curious: can you strip out the doctype string when you are requesting the snippet?

switchtrue Oct 14, 2022

Yes, that's certainly a workaround I will need to consider, but I'd prefer not to have to do this.

It seems strange to me that Astro injects this given that the template is fully customisable, why is this at the discretion of the developer?

For my use case at least, a simple config property to remove (or change) this would suffice.

maxstrebel Dec 1, 2022

+1 to have this as a config option that can be disabled

bjrn · 2022-10-15T12:23:32Z

bjrn
Oct 15, 2022

I’ve solved it for a static build, where we produce regular pages plus some fragment files meant for inclusion. We used a postbuild step that removes the doctype for all files ending with-fragment.html … but it feels strange, and it would be a lot cleaner imho to have a config option to disable auto-inserting the doctype.

If I recall correctly, Astro already has a duplicate-doctype-correction built-in (that is activated when the doctype is encountered but is different from <!DOCTYPE html>? Then that would still make sure there’s no duplicate doctypes in the output.

4 replies

michrome Apr 28, 2023

@bjrn any chance you could share the postbuild script you used?

gittony Apr 28, 2023

+1 on sharing the postbuild if possible.

bjrn Apr 30, 2023

Sure – there are likely more elegant ways to go about it, but using replace in postbuild directly from package.json works as a starting point

"postbuild": "replace \"<!DOCTYPE html>\\s*\" \"\" -r --include=\"*fragment.html\" -q"

Example package.json

thomas-callahan-collibra May 1, 2023

Thanks @bjrn that's a lot simpler than what I was thinking.

natemoo-re · 2023-02-09T20:10:20Z

natemoo-re
Feb 9, 2023
Maintainer

Hey y'all! There was a bit more movement on this in withastro/astro#6084, but we decided to bounce it back to this discussion since it's a larger change coordinated across our compiler and astro itself. We'd love to see a fleshed out proposal for what this feature could look like!

I totally agree that the ability to generate fragments would be very useful, but enabling this would be a breaking change. Do we use a config setting, some file-based pattern like a fragments/ directory, a file extension like .fragment.astro or something else entirely?! Would love to hear feedback from those who are wanting this feature.

2 replies

jordemort Feb 9, 2023

My vote would be for .fragment.astro (or .astrofrag or something if a single one makes it easier for IDE integration) as an extension. I would rather have that than a special directory, because I might want to put fragments anywhere. I'd rather have it than a config setting because it just feels nicer to me to have it act on the extension instead of having to put some magic invocation in each fragment.

ZebraFlesh Feb 9, 2023

From the POV of someone who wants this feature, I'd be happy with either .fragment.astro or a fragments directory that is a sibling to pages. The directory might make the most sense so you can tell what's going on in some random project that you pickup while not requiring any updates to tooling (because they're all still .astro files). But having a new directory with output targets that might be combined with a (possibly empty) pages directory sounds like more work on the compiler side (but I don't know the Astro compiler, so maybe I'm wrong?).

switchtrue · 2023-02-09T21:59:20Z

switchtrue
Feb 9, 2023

A file name .fragment.astro would give more freedom over directory structure.

With our use case we do potentially have a curveball in the form of sitemaps. We have a folder structure like:

src/
├─ pages/
│  ├─ some-page.astro
│  ├─ some-page.head.astro

Both some-page.astro and some-page.head.astro are "fragments". The former gets inserted into the body of a page template and the latter into the head of the same page. We only want one of these to appear in the sitemap so we use the Astro sitemap functionality to generate our sitemaps via a custom method which is roughly:

export default defineConfig({
  ...
  integrations: [react(), sitemap({
    serialize(item) {
      if (item.url.endsWith(".head")) return undefined;
      return getSiteMapEntry(cliConfig.site, item);
    }
  })],
  ...
});

I'm unsure if sitemaps have been considered so far in regards to fragments? I'd be keen to hear to thoughts regarding this.

Arguably this isn't the correct approach as when using fragments there's actually something else at play which ultimately decides which URLs and pages get included where and it should be that thing that generates the sitemaps. But it sure is handy having it in Astro :)

0 replies

bjrn · 2023-02-11T08:52:47Z

bjrn
Feb 11, 2023

For our use case — a micro frontends architecture that allows teams to deploy full pages as well as fragments meant to be used for page composition (e.g. header, footer, various reusable sections and interactive components) — I would still argue that a configuration option, such as the proposed prependDoctype, to be the most logical choice.

it is a corner case; most would would leave it to the default true. (also, Astro already has duplicate doctype prevention in place, so adding doctype in a layout would not create duplicate doctype declarations)
Less magic: prependDoctype: false is what would be expected without knowing Astro, i.e. if I omit the doctype in my layout file, it's not part of the output either.
doctype treatment would become consistent with the treatment of head and body
If a project makes use of fragments, it's highly likely that it's also are using multiple layouts beforehand (and are well aware of the necessity of the <!DOCTYPE html> in the beginning of an html document)
No filename conventions leaking out into the build output (what if someone needs to produce fragments like xyz-partial.html to be compliant with its surroundings)
exclusions from other kinds of outputs (for example the sitemap example earlier in this thread) are still solvable

All that being said, the *.fragment.astro approach would also work.

IMHO the choice comes down to whether Astro wants to endorse using fragments as a design pattern in the same way that component composition is the default today (my vote goes to component composition). Adding a config that allows for disabling the default inclusion of doctype is more subtle than introducing a new filename convention.

For our use case, fragments are used to enable composition between modules that are built using different technologies, and not for internal consumption within the same module. (composition with fragments is handled with ESI – Edge Side Includes).

8 replies

thomas-callahan-collibra Apr 14, 2023

Could it be a similar to getStaticPaths() -- a reserved variable set in the page component defined by Astro meant for use by the generator at build time that you add to control the behavior. Like @bjrn said the default should be compatible with existing code so it's not a breaking change (unless you happen to have used a conflicting variable name). So the default is true so existing behavior is maintained, but if you set it to false, when generating that route the DOCTYPE is omitted.

I'd prefer a page-level option instead of a config because my use case is that I'm exploring having our main site generated statically by Astro, but we have other sites hosted by various vendors and several Wordpress sites that I'd like to have consume a common header, generated by Astro using the same components used to generate the main site's header. So some page components will need to include the DOCTYPE because they're generating regular pages, some shouldn't have it because they're generating snippets. The "Header" component used for both knows its generating a snippet for use across sites, so links include FQDNs, there are no CSS dependencies, etc.

I have this working except that each site consuming the common header would need to strip the DOCTYPE from the fragment HTML returned by the page component, or I'd have to have a proxy that consumes the header fragment, strips the DOCTYPE, and provides it to the consumer sites. Neither is ideal -- the consuming sites aren't all directly under my control so I need to make this as simple as possible to consume, and I'd prefer not to maintain a separate proxy just for this purpose.

bjrn Apr 14, 2023

If I understand your case correctly @thomas-callahan-collibra, wouldn't that be doable by disabling the doctype auto-insertion (given that the prependDoctype config option would exist), and either add a condition in your Layout or a Header component?

thomas-callahan-collibra Apr 14, 2023

So disable it globally in the config, then add the doctype into the templates where you need it? Yes that would work but I'd consider handling it at the page level a little cleaner at least for my use case.

I'll have 8 routes that need it and only one that doesn't. Doing it at the page level almost everything would use the default setup and I'd just turn it off in the one route that needs that, the other way would mean turning it off globally and manually adding it back to 7 routes. And I'd expect my route count to grow over time, so each new route would need to remember to add it in. Yes using a layout file could simplify that but it still feels more complicated.

Honestly though either way is fine, I don't have a super strong opinion.

michrome Apr 28, 2023

+1 for a global prependDoctype setting. Set to true by default (same behaviour as today).

If set to false, it's up to me to ensure a doctype is included in my layouts when I need it.

kireerik May 22, 2023

#266 (comment)

lilnasy · 2023-04-02T17:15:57Z

lilnasy
Apr 2, 2023

I have wanted to use fragments with astro, but I came across more issues than doctype. So if you're using them, I'm curious if you

get flash of unstyled content, and
set X-Robots-Tag: noindex header manually for each fragment.

0 replies

thomas-callahan-collibra · 2023-05-03T19:51:05Z

thomas-callahan-collibra
May 3, 2023

One other important thing I just ran into -- with no <head> or <body>, Astro does not include JS or CSS in the final output, so this solution would need to allow for that. Something like "if <head> and/or <body> is present, insert the appropriate things in the appropriate places, otherwise prepend/append to the output? Or maybe a template directive like insert:head and insert:body to define a location to insert the <script> and <style> tags?

I'm going to include an empty <head> and <body> in my fragment template, then strip those out in the same post-processing I'm going to run to strip out the <DOCTYPE>.

1 reply

thomas-callahan-collibra May 3, 2023

Here's what I came up with in case it's useful to someone else. This is a stripped down version of what I'm using (mine has a bunch of extra stuff in it related to reading specific directories) but the basic idea is you run a Node script after a build that loads any fragment files you want to modify and uses a regex to remove the doctype, head and body tags.

Your component code looks like this:

  <head></head>
  <body>
    <your-astro-components-go-here />
  </body>

The empty <head> and mostly-empty <body> give Astro a way to insert JS and CSS in proper relation to where your Astro HTML will end up in the final document.

Then your build script looks like:

astro build && node clean-production-html.mjs

So it runs a build then sequentially runs a Node script.

And you create clean-production-html.mjs at the root of your project with something like this:

import fs from "fs";
import { fileURLToPath } from "url";
import path, { dirname } from "path";

const __filename = fileURLToPath(import.meta.url);
const __dirname = dirname(__filename);

const filesPath = path.join(__dirname, "dist");

const files = fs.readdirSync(filesPath);
files.forEach((file) => {
  var filePath = path.join(filesPath, file);
  var fileContents = fs.readFileSync(filePath, "utf8");
  fileContents = fileContents.replace(
    /<!DOCTYPE html>|<head>|<\/head>|<body>|<\/body>/g,
    ""
  );
  fs.writeFileSync(filePath, fileContents);
});

You could modify that to scan more folders (my actual code scans specific directories), limit modification to only certain filenames (ones including a ".fragment" modifier using if (file.includes(".fragment")) { ... } ), or whatever. In my case I am also minifying the HTML using html-minifier to remove whitespace before writing it back.

lilnasy · 2023-05-04T19:15:39Z

lilnasy
May 4, 2023

Curious how well this middleware works for you, it removes the doctype (the first 16 bytes) from the response if the request had the header X-Accept: fragment

export async function onRequest ({ request }: { request: Request }, next : () => Promise<Response>) {
    const response = await next()
    if (
        response.ok &&
        response.headers.get('Content-Type') === 'text/html' &&
        request.headers.get('X-Accept') === 'fragment'
    ) {
        const sourceReader = response.body!.getReader()
        let removedDoctype = false

        const fragmentResponseBody = new ReadableStream<Uint8Array>({
            async pull(controller) {
                const { done, value } = await sourceReader.read()
                
                if (done)                controller.close()
                else if (removedDoctype) controller.enqueue(value)
                else                     controller.enqueue(value!.slice(16))
                
                removedDoctype = true
            }
        })

        return new Response(fragmentResponseBody, {
            status     : response.status,
            statusText : response.statusText,
            headers    : response.headers,
        })
    }
}

and you would fetch fragments like so:

const response = await fetch('/', { headers: { 'X-Accept': 'fragment' } })

0 replies

kireerik · 2023-05-22T20:53:04Z

kireerik
May 22, 2023

To keep it simple I created a new proposal based on @michrome's reply (#266 (reply in thread)):

Implement a setting to enable document type assigning #588

0 replies

kireerik · 2023-05-29T07:05:45Z

kireerik
May 29, 2023

Until #588 is implemented here is a simple workaround using an Astro middleware:

export const onRequest = async (context, next) => {
	const response = await next()

	return new Response(
			response.text ?
				(
					await response.text()
				)
					.replace('<!DOCTYPE html>', '')
			:
				response.body
		,
			response
	)
}

Where we do need this document type there we can just add it.

In an Astro component we can use a Fragment component to do so like this:

<Fragment set:html={'<!DOCTYPE HTML>'}/>

7 replies

kireerik May 29, 2023

So here is a less nice but properly working (for the case (as well) where we do need this document type we can add it (properly)) workaround:

const getDocumentType = value => '<!DOCTYPE ' + value + '>'

export const onRequest = async (context, next) => {
	const response = await next()

	return new Response(
			response.text ?
				(await response.text()).replace(
					getDocumentType('html')
					, (() => {
						var result = ''

						const {documentType} = global

						if (documentType) {
							result += getDocumentType(documentType)

							delete global.documentType
						}

						return result
					})()
				)
			:
				response.body
		,
			response
	)
}

Where we do need this document type there we can just add it.

In an Astro component we can do so like this:

---
global.documentType = 'HTML'
---

If someone has a better idea for replacing the usage of global in this workaround please let me know!

lilnasy May 30, 2023

Worth noting that you lose the advantage of astro's streaming architecture this way. This blocks sending anything until everything is rendered.

kireerik May 30, 2023

Yes I agree, thank you so much for noting this!

Right now my middleware setup depends on HTML Minifier Terser which as far as I know doesn't support streaming.

Now I have tried minify-html which is definitely faster using streaming. However the output of it (at least on my index page) is less minified compared to the current solution I am using.

If someone can recommend a better alternative let me know!

Until then I can not use streaming anyway if I am correct.

So once and if this changes I will implement an updated workaround based on your's (@lilnasy) as well and post it here too.

My workaround can also be changed to support streaming if someone needs it like that.

kireerik May 31, 2023

I think it is better to benefit from streaming even in the case of my (middleware) setup.

So here is an updated workaround based on @lilnasy's workaround as well:

const getDocumentType = value => '<!DOCTYPE ' + value + '>'

export const onRequest = async (context, next) => {
	const response = await next()

	return new Response(
			response.text ?
				(() => {
					const reader = response.body.getReader()

					var receivingDocumentType = true
					, addingDocumentType = receivingDocumentType

					return new ReadableStream({
						async pull(controller) {
							const {
								done, value
							} =
								await reader.read()

							if (done)
								controller.close()
							else
								if (receivingDocumentType)
									receivingDocumentType = false
								else {
									if (addingDocumentType) {
										const {documentType} = global

										if (documentType) {
											controller.enqueue(
												Buffer.from(
													getDocumentType(
														documentType
													)
												)
											)

											delete global.documentType
										}

										addingDocumentType =
											receivingDocumentType
									}

									controller.enqueue(value)
								}
						}
					})
				})()
			:
				response.body
		,
			response
	)
}

Where we do need this document type there we can just add it.

In an Astro component we can do so like this:

---
global.documentType = 'HTML'
---

If someone has a better idea for replacing the usage of global in this workaround please let me know!

kireerik May 31, 2023

The above updated workaround only works properly if we define global.documentType in a component at last that already returns something. Otherwise we would gather the parts until we detect global.documentType and then push the gathered parts and the rest as they are coming in (if there is some still remaining at all (stream reading ended if not)).

tropperstyle · 2023-05-31T04:54:15Z

tropperstyle
May 31, 2023

If someone has a better idea for replacing the usage of global in this workaround please let me know!

I suggested middleware access to the matched route module in #555 (comment) but it didn't gain any traction

I'm on the outside looking in, but digging deeper it looks like all the invocations of callMiddleware already have mod available in local scope, so it could just be a matter of passing that through.

1 reply

kireerik May 31, 2023

Thank you! What is mod? How can we pass it through?

I have tried a couple of ways based on your comment (#555 (comment)), but the exported constant is not going through.

kireerik · 2023-06-17T03:15:51Z

kireerik
Jun 17, 2023

Eventually I preferred using (static) Astro (page) (file) endpoints (instead of (Astro) components):

refo

This completely eliminates this issue.

1 reply

michrome Aug 27, 2023

@matthewp this describes my needs here. When using HTMX I want a RESTful endpoint that returns a snippet of HTML: not JSON and not a complete/valid HTML document.

Because of that, I want to define the endpoint using Astro component syntax, not a body defined as JSON.

matthewp · 2023-08-25T12:40:14Z

matthewp
Aug 25, 2023
Maintainer

Random idea I had about this, not sure if good or not: if a page ends with .fragment.astro we do not do any page injections (no doctype, no link/script injection). For ex: src/pages/foo/bar.fragment.astro. Instead this would just be the unmodified HTML produced by this route.

We could still provide the scripts/links via a Header. It would require that the frontend making the request does something with the header. I looked into htmx and I'm not sure that they have an API for providing styles/scripts in this way.

11 replies

beckerei Aug 25, 2023

I think it should be possible (somehow) to control this by the user.
For some cases you don't want or need the styles, in my case I rely on the styles to be included within the fragment. I'd prefer styles inline.

At the moment I have another astro page, which renders the same component but was a full page. I would prefer styles not to be rendered inline there, but it is not possible to configure inlineStylesheets per page/path.

I also think we have uncommon (enterprise) use-case.
The component I'm talking about is our global navigation, which is injected into e.g. Rails, Next and SPAs. The page containing the navigation is served within a frame-set (yes, frame-set :()

But I think this gives some perspective on what astro is already capable off, and what real cases exists.

matthewp Aug 26, 2023
Maintainer

Ok, so it looks like HTMX has a hook afterSwap: https://htmx.org/events/#htmx:afterSwap

Assuming this fires synchronously, you could remove the scripts and styles using that hook and put them into the head, probably diffing to prevent duplicates.

Perhaps fragments should always have their stylesheets inlined. Not 100% on that, but something to investigate.

This is all seeming more clear to me now, and is doable.

lilnasy Aug 26, 2023

I think the decision (or a hook into the decision) should be up to the user. For example, they might want to fetch the newly required styles themselves before appending to DOM.

ZebraFlesh Aug 26, 2023

Agree that the decision should be up to the user. This is very much an “it depends” scenario when choosing the “right” answer.

matthewp Aug 28, 2023
Maintainer

Yeah, my preference would be that Astro leaves the scripts/styles in. If you are using Tailwind or other global styles, and not using hoisted scripts, you might not run into this issue. I just want to make sure that it's possible for an integration or something to solve the problem.

matthewp · 2023-09-13T16:36:10Z

matthewp
Sep 13, 2023
Maintainer

This feature has been accepted to Stage 2 and will be worked on by someone on the core team over the next few months. Please see the Stage 2 proposal here; a Stage 3 proposal is in the works which will include the API design.

0 replies

ruohki · 2023-09-26T17:42:25Z

ruohki
Sep 26, 2023

Would it be possible to enable opting out of all generated tags like <head> <body> <html> etc? This would benefit frameworks like htmx alot. I know there is a middleware solution, but i find that rather hacky. and a .fragment.astro would just perfectly fit this usecase as well.

---
Astro.generator.setGenerateInlineTags(false);

---

or something

1 reply

matthewp Sep 27, 2023
Maintainer

Please leave that feedback here: #697

A way to not use <!DOCTYPE html> #266

FredKSchott Aug 21, 2022 Maintainer

Replies: 16 comments · 39 replies

FredKSchott Oct 14, 2022 Maintainer Author

natemoo-re Feb 9, 2023 Maintainer

matthewp Aug 25, 2023 Maintainer

matthewp Aug 26, 2023 Maintainer

matthewp Aug 28, 2023 Maintainer

matthewp Sep 13, 2023 Maintainer

matthewp Sep 27, 2023 Maintainer

FredKSchott
Aug 21, 2022
Maintainer

Replies: 16 comments 39 replies

FredKSchott Oct 14, 2022
Maintainer Author

natemoo-re
Feb 9, 2023
Maintainer

matthewp
Aug 25, 2023
Maintainer

matthewp Aug 26, 2023
Maintainer

matthewp Aug 28, 2023
Maintainer

matthewp
Sep 13, 2023
Maintainer

matthewp Sep 27, 2023
Maintainer