🐛 BUG: Astro glob doesn't work when there is a variable in a template string

[zellwk]

What version of astro are you using?

1.0.0-beta.31

Are you using an SSR adapter? If so, which one?

no

What package manager are you using?

npm

What operating system are you using?

Mac

Describe the Bug

You cannot use a template strong that contains a variable to glob for content.

---
const path = 'blog'
const posts1 = await Astro.glob(`./${path}/*.md`) // Doesn't work. No matches found. 
const posts2 = await Astro.glob(`./blog/*.md`) // Returns a list of posts. 

console.log(posts1)
console.log(posts2)
---

Use Case:

I want this feature to work for dynamically generated pages. For example:

---
export function getStaticPaths() {
  return [
    { params: { contentType: 'blog' } },
    { params: { contentType: 'conversations' } },
    { params: { contentType: 'videos' } },
  ];
}

const { contentType } = Astro.params
const posts1 = await Astro.glob(`./${contentType}/*.md`)
---

Potential problem area

Upon further research, I noticed you can't pass variables into Astro.glob. Not sure if this is related.

---
const path = `./blog/*.md`
const posts3 = await Astro.glob(path) // Astro.glob() and import.meta.glob() can only accept string literals.
---

Link to Minimal Reproducible Example

https://stackblitz.com/edit/github-omfzjm?file=src/pages/index.astro

Participation

• I am willing to submit a pull request for this issue.

[retronav]

Hi, this behavior of Astro.glob is intended because it uses Vite's import.meta.glob under the hood. And from its documentation:

You should also be aware that glob imports do not accept variables, you need to directly pass the string pattern.

I think its that the code is statically parsed, turned into an AST, and then the glob is computed from the AST, which is why string literals don't work (I maybe wrong though). That's why you will have to use a static string glob for Astro.glob.
Related issues: vitejs/vite#5478, vitejs/vite#5478 (comment)

[mash-graz]

it's indeed a very unpleasant limitation, which IMHO shouldn't be present in a mainly path-structured SSG.

it's a very annoying hindrance, if you have to handle nested folder hierarchies of content, where you do not know the exact structure and names in advance. as a workaround you always have to fetch the whole tree and handle the selection/filtering by your on custom routines, because astro doesn't provide any useful mechanism beside this really stupid glob-"feature", which is utterly inadequate for a wide range of practical cases. :(

[zellwk]

@retronav @mash-graz Completely understand.

Actually it's not THAT unpleasant of a limitation. I found a workaround — which is to glob everything I need, then filter afterwards.

---
export function getStaticPaths() {
  return [
    { params: { contentType: 'blog' } },
    { params: { contentType: 'conversations' } },
    { params: { contentType: 'videos' } },
  ];
}

const { contentType } = Astro.params
const posts = await Astro.glob(`./**/*.md`)
const filtered = posts.filter(post => {
  return post.url.includes(contentType)
})
---

Maybe we can put this solution into the docs?

[mash-graz]

Actually it's not THAT unpleasant of a limitation. I found a workaround...

It depends on the requirements of the actual usage scenarios, if a solution will fulfil the given demands.

if you only want to find the actual path of some dependency or component, like in the case of a build system (vite, etc.), the glob solution may look sufficient, but in a SSG solution, which tries to give the user as much freedom and flexibility as possible, it's IMHO just a HORRIBLE LIMITATION and an unacceptable bad joke and by no means sufficient.

sure, i use the already mentioned workaround, too. nevertheless, it's a very inefficient and annoying workaround, if you e.g. have to migrate a rather huge web-shop with tons of categories and articles in nested file hierarchies resp. handle access to this content in a simple and efficient manner. well, it works somehow, but i definitely would describe it as desirable or exemplary.

why should we gloss over such a significant flaw of software instead of simply criticizing its actual deficiencies?

i'm rather sure, that some users will even try to bypass this actual limitations by utilizing nodes file system access routines as another possible but highly incompatible "workaround", just because there isn't any simple and evident better solution for this very common kind of more flexible directory-like access provided by astro out of the box until now.

[zellwk]

I understand your point of view. Our tools should work without limitations and workarounds as much as possible. I'll look forward to a better and more efficient solution in the future! But for now, I'll continue using the workaround.

Thanks for your work in this project.

[tony-sull]

As mentioned above this is expected behavior, but I 100% agree it's something that would be nice to support in the future!

We're a bit limited by how Vite's import.meta.glob() works here as long as Astro.glob() builds on top of that, here's the Vite documentation calling out the lack of support for variables. I believe one of Astro's open docs PRs also adds more detail here, making it clear that this isn't currently supported

The Vite team is currently working on version 3 and have an open Discussion thread on their repo for feature requests and concerns. This would be a great one to bring up over there as well, I haven't actually seen it come up in their discussion thread yet!

I'm going to close this issue out as this is a known and expected limitation, but that doesn't mean we don't want to support the use case long term! Since it would be new functionality that also might be a good time to start a discussion on our own RFC repo. It's always possible that there's a bit of magic we could build into Astro.glob() to support this even if Vite's API doesn't get the behavior 👍

[sahil1sk]

The above discuss all correct. But I also find another thing for me this error because of Incorrect upper folder structure.
// To use Astro.glob function upper folder structure should be clean like no upper folder should be like contains "Folder name (anything)" these "()" remove this circular brackets from folders name if you having any.

[bluwy]

@sahil1sk that was actually a bug in Vite that should be fixed now. We will be bumping the Vite version Astro uses soon, or you can use npm overrides to use that at the meantime.

[cangSDARM]

Any update?

[Its-Just-Nans]

@zellwk #3418 (comment)

You solution with filter is working but not for every file type

[zellwk]

@Its-Just-Nans You don't need this anymore. Use Content Collections instead.

@zellwk #3418 (comment)

You solution with filter is working but not for every file type

[Its-Just-Nans]

Nop because I want the folder to be external

[mash-graz]

You solution with filter is working but not for every file type

Concerning dynamic path and import handling in astro you will always face rather frustrating limitations.

Most of them are an unavoidable result of this rather questionable strict adoption of usage paradigms from vite. While they may fit very well for most TS/JS development work and code optimization, they are not always optimal for flexible HTML and scripted content processing. That's the main reason, why only those operations, which define values in the frontmater in a similar way as static ESM imports will work reliable in astro, but all other more dynamic kinds of access or definition/processing of parameters in the body -- like a simple Image src='...'-attribute inside the page contend instead of an import directive in the preamble -- will not work very well in astro. This affects the whole plugin system and any other kind of script controlled path handling.

yes -- There are plausible reasons to handle the page creation in this manner! It's indeed very fast in those cases, where you don't have to find workarounds for its limitations, but the mentioned predefined constraints are unfortunately rather strong.

It's IMHO one of the most serious architectural drawback of astros fundamental design.

And although the astro is otherwise a really nice solution with fantastic component syntax, lots of useful features and third party support, I'm personally more and more inclined to use deno and the lume SSG instead, just because of this particular design flaw. The plugin system and path handling in lume looks much more transparent and development friendly compared to astro and vite concerning this hidden constraints if you are more focused on content oriented HTML preprocessing, and not only looking for a complex JS/TS build system and JS related bundle optimization.