Listing page tries to list documents disabled in _quarto.yml

[royfrancis]

Description

When a qmd is disabled in _quarto.yml, the listing page still tries to list it with this warning:

WARN: File listing/test.qmd in the listing 'listing' contains no metadata.

And the listing page looks a bit broken. The disabled post is rendered on the listing page as an empty bordered container with no content.

▸ quarto --version
1.9.38

More details:
I have a directory setup like this:

.
├── _quarto.yml
├── index.qmd
├── listing.qmd
└── test.qmd

_quarto.yml

project: 
  type: website
  render: 
    - "*.qmd"
    - "!test.qmd"

listing.qmd

---
title: "Listing"
listing:
  type: grid
  contents:
    - "*.qmd"
---

index.qmd

---
title: "Home"
---

This is the home page.

test.qmd

---
title: "Test page"
---

This is a test page.

[mcanouil]

Important

Accessibility: To improve accessibility, please add alternative text to your screenshots. This helps all users, including those using screen readers, to understand the context of the images. A brief description can make a big difference! See Good Alt Text, Bad Alt Text — Making Your Content Perceivable.

[mcanouil]

Indeed, the two globs does work independently:

1. Quarto resolves the project's own input file list (correct)

   Before any page renders, projectInputFilesInternal() walks render:, splitting !-prefixed entries into excludes, and produces project.files.input — this list correctly omits test.qmd.

   • quarto-cli/src/project/project-context.ts

     Lines 893 to 1009 in ab395f6

     	async function projectInputFilesInternal(
     	project: ProjectContext,
     	metadata?: ProjectConfig,
     	): Promise<{ files: string[]; engines: string[] }> {
     	// Resolve engines so engineIgnoreDirs() uses all engines (including external)
     	await resolveEngines(project);
     	const { dir } = project;
     	const outputDir = metadata?.project[kProjectOutputDir];
     	// Ignore project standard and hidden files
     	const projIgnoreGlobs = projectHiddenIgnoreGlob(dir);
     	// map to regex
     	const projectIgnores = projIgnoreGlobs.map((glob) =>
     	globToRegExp(glob, { extended: true, globstar: true })
     	);
     	type FileInclusion = {
     	file: string;
     	engineName: string;
     	engineIntermediates: string[];
     	};
     	const addFile = async (file: string): Promise<FileInclusion[]> => {
     	// ignore the file if it is in the output directory
     	if (
     	outputDir &&
     	ensureTrailingSlash(dirname(file)).startsWith(
     	ensureTrailingSlash(join(dir, outputDir)),
     	) &&
     	ensureTrailingSlash(join(dir, outputDir)).startsWith(
     	ensureTrailingSlash(dir),
     	)
     	) {
     	return [];
     	}
     	const engine = await fileExecutionEngine(file, undefined, project);
     	// ignore the file if there's no engine to handle it
     	if (!engine) {
     	return [];
     	}
     	const engineIntermediates = executionEngineIntermediateFiles(
     	engine,
     	file,
     	);
     	return [{
     	file,
     	engineName: engine.name,
     	engineIntermediates: engineIntermediates,
     	}];
     	};
     	const addDir = async (dir: string): Promise<FileInclusion[]> => {
     	const promises: Promise<FileInclusion[]>[] = [];
     	for await (
     	const walkEntry of walk(dir, {
     	includeDirs: false,
     	// this was done b/c some directories e.g. renv/packrat and potentially python
     	// virtualenvs include symblinks to R or Python libraries that are in turn
     	// circular. much safer to not follow symlinks!
     	followSymlinks: false,
     	skip: [kSkipHidden].concat(
     	engineIgnoreDirs().map((ignore) =>
     	globToRegExp(join(dir, ignore) + SEP)
     	),
     	),
     	})
     	) {
     	const pathRelative = pathWithForwardSlashes(
     	relative(dir, walkEntry.path),
     	);
     	if (projectIgnores.some((regex) => regex.test(pathRelative))) {
     	continue;
     	}
     	promises.push(addFile(walkEntry.path));
     	}
     	const inclusions = await Promise.all(promises);
     	return inclusions.flat();
     	};
     	const addEntry = async (entry: string) => {
     	if (Deno.statSync(entry).isDirectory) {
     	return addDir(entry);
     	} else {
     	return addFile(entry);
     	}
     	};
     	const renderFiles = metadata?.project[kProjectRender];
     	let inclusions: FileInclusion[];
     	if (renderFiles) {
     	const exclude = projIgnoreGlobs.concat(outputDir ? [outputDir] : []);
     	const resolved = resolvePathGlobs(dir, renderFiles, exclude, {
     	mode: "auto",
     	});
     	const toInclude = ld.difference(
     	resolved.include,
     	resolved.exclude,
     	) as string[];
     	inclusions = (await Promise.all(toInclude.map(addEntry))).flat();
     	} else {
     	inclusions = await addDir(dir);
     	}
     	const files = inclusions.map((inclusion) => inclusion.file);
     	const engines = ld.uniq(inclusions.map((inclusion) => inclusion.engineName));
     	const intermediateFiles = inclusions.map((inclusion) =>
     	inclusion.engineIntermediates
     	).flat();
     	const inputFiles = Array.from(
     	new Set(files).difference(new Set(intermediateFiles)),
     	);
     	return { files: inputFiles, engines };
     	}

   Underlying glob engine used here:

   • quarto-cli/src/core/path.ts

     Lines 137 to 197 in ab395f6

     	export function filterPaths(
     	root: string,
     	paths: string[],
     	globs: string[],
     	options?: GlobOptions,
     	): ResolvedPathGlobs {
     	// filter the list of globs
     	// using the paths
     	const expandGlobs = (targetGlobs: string[]) => {
     	const expanded: string[] = [];
     	for (const glob of targetGlobs) {
     	const needSlash = !root.endsWith("/") && !glob.startsWith("/");
     	const regex = globToRegExp(`${root}${needSlash ? "/" : ""}${glob}`);
     	const matchingFiles = paths.filter((path) => {
     	return regex.test(path);
     	});
     	expanded.push(...matchingFiles);
     	}
     	return ld.uniq(expanded);
     	};
     	return resolveGlobs(root, globs, expandGlobs, options);
     	}
     	export interface GlobOptions {
     	mode: "strict" | "auto" | "always";
     	explicitSubfolderSearch?: boolean; // set this to true to never prepend `**/` to globs to create nested directory searching
     	}
     	export function resolvePathGlobs(
     	root: string,
     	globs: string[],
     	exclude: string[],
     	options?: GlobOptions,
     	): ResolvedPathGlobs {
     	// expand a set of globs
     	const expandGlobs = (targetGlobs: string[]) => {
     	const expanded: string[] = [];
     	for (const glob of targetGlobs) {
     	try {
     	for (
     	const file of expandGlobSync(
     	glob,
     	{ root, exclude, includeDirs: true, extended: true, globstar: true },
     	)
     	) {
     	expanded.push(file.path);
     	}
     	} catch (e) {
     	// expandGlobSync can throw NotFound if a file is deleted between
     	// directory listing and stat (TOCTOU race). This is expected during
     	// preview when the IDE or other processes modify the project
     	// directory concurrently.
     	if (!(e instanceof Deno.errors.NotFound)) {
     	throw e;
     	}
     	}
     	}
     	return ld.uniq(expanded);
     	};
     	return resolveGlobs(root, globs, expandGlobs, options);
     	}

   At this point in the workflow, everything is still correct: test.qmd is not in project.files.input.

2. listing.qmd renders, contents: ["*.qmd"] gets resolved

   readContents() is invoked for the listing page, and hands the glob to filterListingFiles().

   • quarto-cli/src/project/types/website/listing/website-listing-read.ts

     Lines 700 to 781 in ab395f6

     	async function readContents(
     	source: string,
     	project: ProjectContext,
     	listing: ListingDehydrated,
     	) {
     	debug(`[listing] Reading listing '${listing.id}' from ${source}`);
     	debug(`[listing] Contents: ${
     	listing.contents.map((lst) => {
     	return typeof lst === "string" ? lst : "<yaml>";
     	}).join(",")
     	}`);
     	const listingItems: ListingItem[] = [];
     	const listingItemSources = new Set<ListingItemSource>();
     	// Accumulate the files that would be allowed to be included
     	// This will include:
     	// - project input files
     	const inputsWithoutSource = project.files.input.filter((file) =>
     	file !== source
     	);
     	const filterListingFiles = (globOrPaths: string[]) => {
     	const hasDefaultGlob = globOrPaths.some((glob) => {
     	return glob === kDefaultContentsGlobToken;
     	});
     	const resolvedGlobs = globOrPaths.map((glob) => {
     	if (glob === kDefaultContentsGlobToken) {
     	return "*";
     	} else {
     	return glob;
     	}
     	});
     	const expandedGlobs = resolvedGlobs.map((resolved) => {
     	return expandGlob(source, project, resolved);
     	});
     	const hasInputs = expandedGlobs.some((expanded) => {
     	return !!expanded.inputs;
     	});
     	const finalGlobs = expandedGlobs.map((expanded) => {
     	return expanded.glob;
     	});
     	// push a glob excluding the project output directory
     	const outDir = project.config?.project["output-dir"];
     	if (outDir) {
     	finalGlobs.push(`!/${outDir}/**`);
     	}
     	// Convert a bare directory path into a consumer
     	// of everything in the directory
     	if (hasInputs || hasDefaultGlob) {
     	// If this is a glob, expand it
     	const filtered = filterPaths(
     	dirname(source),
     	inputsWithoutSource,
     	finalGlobs,
     	);
     	// If a glob points to a file that exists, go ahead and just include
     	// that, even if it isn't a project input
     	for (const glob of finalGlobs) {
     	if (safeExistsSync(glob) && Deno.statSync(glob).isFile) {
     	filtered.include.push(
     	isAbsolute(glob) ? glob : join(Deno.cwd(), glob),
     	);
     	}
     	}
     	return filtered;
     	} else {
     	return resolvePathGlobs(
     	dirname(source),
     	finalGlobs,
     	["_*", ".*", "**/_*", "**/.*", source],
     	);
     	}
     	};

3. Branch decision: is the glob a directory?

   expandGlob() checks whether the glob string resolves to an existing directory on disk.

   • quarto-cli/src/project/types/website/listing/website-listing-read.ts

     Lines 1427 to 1458 in ab395f6

     	function expandGlob(
     	source: string,
     	project: ProjectContext,
     	globOrPath: string,
     	) {
     	const getPath = () => {
     	if (globOrPath.startsWith("/")) {
     	return join(project.dir, globOrPath);
     	} else {
     	const sourcePath = dirname(source);
     	return join(sourcePath, globOrPath);
     	}
     	};
     	// If the glob resolves to a directory, we will
     	// interpret this as meaning that we should glob _inputs_
     	// from that directory (rather than all files, for example)
     	//
     	// If the glob resolves to a non-directory path
     	// we will treat it as globbing against the file system,
     	// not limiting itself to considering project inputs
     	const globOrPathAsPath = getPath();
     	try {
     	if (Deno.statSync(globOrPathAsPath).isDirectory) {
     	return { glob: join(globOrPath, "**"), inputs: true };
     	} else {
     	return { glob: globOrPath, inputs: false };
     	}
     	} catch {
     	return { glob: globOrPath, inputs: false };
     	}
     	}

   "*.qmd" is not a directory -> hasInputs = false, hasDefaultGlob = false -> workflow falls into the unsafe branch.

   (Had contents: instead been a directory, e.g., "posts/", or omitted entirely, the workflow would take the safe branch, matching only against project.files.input, no bug there.)

4. Unsafe branch globs the raw filesystem, bypassing Step 1's result entirely

   Same function, else branch: calls resolvePathGlobs() directly against disk.
   project.files.input (the correct, already-computed exclusion list from step 1) is never consulted here.

   • quarto-cli/src/project/types/website/listing/website-listing-read.ts

     Lines 700 to 781 in ab395f6

     	async function readContents(
     	source: string,
     	project: ProjectContext,
     	listing: ListingDehydrated,
     	) {
     	debug(`[listing] Reading listing '${listing.id}' from ${source}`);
     	debug(`[listing] Contents: ${
     	listing.contents.map((lst) => {
     	return typeof lst === "string" ? lst : "<yaml>";
     	}).join(",")
     	}`);
     	const listingItems: ListingItem[] = [];
     	const listingItemSources = new Set<ListingItemSource>();
     	// Accumulate the files that would be allowed to be included
     	// This will include:
     	// - project input files
     	const inputsWithoutSource = project.files.input.filter((file) =>
     	file !== source
     	);
     	const filterListingFiles = (globOrPaths: string[]) => {
     	const hasDefaultGlob = globOrPaths.some((glob) => {
     	return glob === kDefaultContentsGlobToken;
     	});
     	const resolvedGlobs = globOrPaths.map((glob) => {
     	if (glob === kDefaultContentsGlobToken) {
     	return "*";
     	} else {
     	return glob;
     	}
     	});
     	const expandedGlobs = resolvedGlobs.map((resolved) => {
     	return expandGlob(source, project, resolved);
     	});
     	const hasInputs = expandedGlobs.some((expanded) => {
     	return !!expanded.inputs;
     	});
     	const finalGlobs = expandedGlobs.map((expanded) => {
     	return expanded.glob;
     	});
     	// push a glob excluding the project output directory
     	const outDir = project.config?.project["output-dir"];
     	if (outDir) {
     	finalGlobs.push(`!/${outDir}/**`);
     	}
     	// Convert a bare directory path into a consumer
     	// of everything in the directory
     	if (hasInputs || hasDefaultGlob) {
     	// If this is a glob, expand it
     	const filtered = filterPaths(
     	dirname(source),
     	inputsWithoutSource,
     	finalGlobs,
     	);
     	// If a glob points to a file that exists, go ahead and just include
     	// that, even if it isn't a project input
     	for (const glob of finalGlobs) {
     	if (safeExistsSync(glob) && Deno.statSync(glob).isFile) {
     	filtered.include.push(
     	isAbsolute(glob) ? glob : join(Deno.cwd(), glob),
     	);
     	}
     	}
     	return filtered;
     	} else {
     	return resolvePathGlobs(
     	dirname(source),
     	finalGlobs,
     	["_*", ".*", "**/_*", "**/.*", source],
     	);
     	}
     	};

   Result: test.qmd matches "*.qmd" on disk and is added to the listing's candidate file list, contradicting Step 1.

5. Candidate file gets turned into a listing item, metadata lookup fails

listItemFromFile() looks up test.qmd's metadata via the project's input index, but since test.qmd was never in project.files.input, no metadata exists.
This triggers the warning and produces an item with blank title/date/description.

• quarto-cli/src/project/types/website/listing/website-listing-read.ts

  Lines 1071 to 1209 in ab395f6

  	async function listItemFromFile(
  	input: string,
  	project: ProjectContext,
  	listing: ListingDehydrated,
  	) {
  	if (!isAbsolute(input)) {
  	throw new InternalError(`input path ${input} must be absolute.`);
  	}
  	const projectRelativePath = relative(project.dir, input);
  	const target = await inputTargetIndex(
  	project,
  	pathWithForwardSlashes(projectRelativePath),
  	);
  	const inputTarget = await resolveInputTarget(
  	project,
  	pathWithForwardSlashes(projectRelativePath),
  	false,
  	);
  	const docRawMetadata = target?.markdown.yaml;
  	const directoryMetadata = await directoryMetadataForInputFile(
  	project,
  	dirname(input),
  	);
  	const documentMeta = mergeConfigs(
  	project.config,
  	directoryMetadata,
  	docRawMetadata,
  	) as Metadata;
  	// Consult the website project draft list directly here
  	// since this is occuring before the input index
  	// is populated.
  	const projectDraft = isProjectDraft(projectRelativePath, project);
  	const draftMode = projectDraftMode(project);
  	if (!isDraftVisible(draftMode) && (documentMeta?.draft || projectDraft)) {
  	// This is a draft, don't include it in the listing
  	return undefined;
  	} else {
  	if (
  	!docRawMetadata && (extname(input) === ".qmd" ||
  	extname(input) === ".ipynb")
  	) {
  	warning(
  	`File ${input} in the listing '${listing.id}' contains no metadata.`,
  	);
  	}
  	// See if we have a max desc length
  	const maxDescLength = listing[kMaxDescLength] as number ||
  	kDefaultMaxDescLength;
  	// Create the item
  	const filename = basename(projectRelativePath);
  	const filemodified = fileModifiedDate(input);
  	const description = documentMeta?.description as string ||
  	documentMeta?.abstract as string ||
  	descriptionPlaceholder(inputTarget?.outputHref, maxDescLength);
  	const imageRaw = documentMeta?.image as string | boolean;
  	const image = imageRaw !== undefined && typeof imageRaw === "string"
  	? pathWithForwardSlashes(
  	listingItemHref(imageRaw, dirname(projectRelativePath)),
  	)
  	: undefined;
  	const imageAlt = documentMeta?.[kImageAlt] as string | undefined;
  	const imageLazyLoading = documentMeta?.[kImageLazyLoading] as
  	| boolean
  	| undefined;
  	const date = documentMeta?.date
  	? parsePandocDate(resolveDate(input, documentMeta?.date) as string)
  	: undefined;
  	const datemodified = documentMeta?.date
  	? parsePandocDate(
  	resolveDate(input, documentMeta?.[kDateModified]) as string,
  	)
  	: undefined;
  	const authors = parseAuthor(documentMeta?.author);
  	let structuredAuthors;
  	if (authors) {
  	structuredAuthors = cslNames(
  	authors?.filter((auth) => auth !== undefined).map((auth) => auth?.name),
  	);
  	}
  	const author = structuredAuthors
  	? uniq(
  	structuredAuthors.map((auth) =>
  	auth.literal || `${auth.given} ${auth.family}`
  	),
  	)
  	: [];
  	const readingContext = target?.markdown?.markdown
  	? estimateReadingTimeMinutes(target.markdown.markdown)
  	: undefined;
  	let readingtime = undefined;
  	let wordcount = undefined;
  	if (readingContext) {
  	readingtime = readingContext.readingTime;
  	wordcount = readingContext.wordCount;
  	}
  	const categories = documentMeta?.categories
  	? Array.isArray(documentMeta?.categories)
  	? documentMeta?.categories
  	: [documentMeta?.categories]
  	: undefined;
  	const item: ListingItem = {
  	...documentMeta,
  	path: `/${projectRelativePath}`,
  	outputHref: inputTarget?.outputHref,
  	[kFieldTitle]: target?.title,
  	[kFieldDate]: date,
  	[kFieldDateModified]: datemodified,
  	[kFieldAuthor]: author,
  	[kFieldCategories]: categories,
  	[kFieldImage]: image,
  	[kFieldImageAlt]: imageAlt,
  	[kFieldImageLazyLoading]: imageLazyLoading,
  	[kFieldDescription]: description,
  	[kFieldFileName]: filename,
  	[kFieldFileModified]: filemodified,
  	[kFieldReadingTime]: readingtime,
  	[kFieldWordCount]: wordcount,
  	};
  	return {
  	item,
  	source: target !== undefined
  	? ListingItemSource.document
  	: ListingItemSource.rawfile,
  	};
  	}
  	}

---

It should be possible and relatively cheap I guess to add an additional check, using the helpers already there, to ensure ensure the listing files are also a member of project.files.input.

[mcanouil]

cc @cderv I believe this is worth fixing.

[cderv]

Let's open an issue with a reproducible example and the detailled info you found. I'll have a look. Thank you !

[cderv]

Or maybe I can from this dscusson - let me try

[mcanouil]

• Listing page tries to list documents disabled in _quarto.yml #14648