render_book("filename") complains about missing "index.Rmd"

[otoomet]

I have a project in a separate folder that contains three files (and only these three files):

foo.rmd`

# Chapter Foo

bar.rmd

# Chapter bar

and _bookdown.yml:

book_filename: foobar

rmd_files: ["foo.rmd", "bar.rmd"]

I attempt to knit the book as

Rscript -e "bookdown::render_book('foo.rmd')"

This gives me an error

Error in stop_if_not_exists("index.Rmd") : 
  Some files were not found: index.Rmd
Calls: <Anonymous> -> stop_if_not_exists
Execution halted

So bookdown is looking for index.Rmd although this file is nowhere in my project, and ?render_book states that

...
input: A directory, an input filename or multiple filenames. For a
directory, ‘index.Rmd’ will be used if it exists in this
(book) project directory. For filenames, if ‘preview = TRUE’,
only files specified in this argument are rendered, otherwise
all R Markdown files specified by the book are rendered.

This occurs in R 4.2, bookdown 0.27 and also the github version 0.27.1. It worked in R 4.1, bookdown 0.24.

For a larger project a workaround seems to be to create an empty index.Rmd file, but not for this simple example.

> xfun::session_info()
R version 4.2.0 (2022-04-22)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 20.04.4 LTS

Locale:
  LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
  LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
  LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

Package version:
  compiler_4.2.0  graphics_4.2.0  grDevices_4.2.0 parallel_4.2.0  stats_4.2.0     tools_4.2.0     utils_4.2.0     xfun_0.31      

[cderv]

I had another report of this recently in https://community.rstudio.com/t/render-book-error-stop-if-not-exists-and-some-files-were-not-found/139927/19

I don't think it comes from the same code part, but change impact this is the same. In bookdown 0.27 we enforced the checking of index.Rmd files that are expected in a bookdown project. This is where the YAML header for the book metadata are stored.

We were not checking for index.Rmd before but assume it was there. Some feature were not working well before because of that (auto guessing output format) as we introduced the check to fix #1332

I think the error is thrown for you from

bookdown/R/render.R

Lines 79 to 87 in 7e9b46f

	} else if (is.null(output_format) || is.character(output_format)) {
	if (is.null(output_format) || identical(output_format, 'all')) {
	# formats can safely be guess when considering index.Rmd and its expected frontmatter
	# and not another Rmd file which has no expected YAML frontmatter
	stop_if_not_exists("index.Rmd")
	all_formats = rmarkdown::all_output_formats("index.Rmd")
	# when no format provided, return name of the first resolved
	output_format = if (is.null(output_format)) all_formats[[1]] else all_formats
	}

from this part, and is necessary so that we can correctly detect which format to use (merging all format from _output.yaml and index.Rmd.

It seems bookdown was used without a index.Rmd file without much issue. So maybe we need to reconsider and be less strict about this file existence.

Where are you format defined if not in a index.Rmd file ? Do you have a YAML header in foo.rmd or another file ? All our template and example are using a index.Rmd which is always the first file of a project.

@yihui what is you take on this ?
Was index.Rmd always assumed to be in the project ? I thought so bu maybe not.
Otherwise, we were not aware of usage without this index.Rmd file.

Thanks.

[otoomet]

@cderv , was it a question to me?

• I never tried the "foo" example with bookdown 0.24. So I cannot tell if it works (sorry, overlooked it in the bug report).
• In my actual project, I have index.rmd (not lower case "r"), and so far Rscript -e "bookdown::render_book('index.rmd')" worked. Now I also created an empty index.Rmd in there, and it seems to work.
• I am not 100% sure what do you mean as "output format", I have _bookdown.yml, that defines some of the ouput format parameters. I assumed that html is the default output format:

book_filename: machinelearning-py
output_dir: build
delete_merged_file: true
language:
  ui:
    chapter_name: ["Chapter "]
new_session: yes

rmd_files: ["index.md",
"python.md",
"numpy-pandas.md",
"plotting.md",
...

(Note: the files are *.md, not *.rmd, see #1262 for explanations)

• The documentation (cited above) mentions index.Rmd as the default file, but it seems like one can use any other file instead of it.

[cderv]

In my actual project, I have index.rmd (not lower case "r"), and so far Rscript -e "bookdown::render_book('index.rmd')" worked. Now I also created an empty index.Rmd in there, and it seems to work.

So this confirms it works with index.Rmd files, and if this file is missing we are throwing an error. This is expected behavior currently as we are using index.Rmd to scan YAML header for a possible output field.

Usually in a bookdown project, there is a main file with some YAML header. I was wondering which one you used if not index.Rmd

[otoomet]

In my big project I was using index.rmd (all lower case, sorry for the typo above), that one has the yaml header.

It seems that index.Rmd is currently required. But my reading from the docs is that index.Rmd is just the default starting point for directories, but if one submits another file name then it is not needed. This is how I understood the docs, and up to now it also worked in this way:

an input filename or multiple filenames. For a directory, ‘index.Rmd’ will be used if it exists in this (book) project directory

I'd expect it either to a) let the user choose an arbitrary file name, and build the book based on that file and the config file; or b) state clearly that index.Rmd is needed (and capitalization matters).

I'd prefer the option a), there may be various reasons why one cannot use index.Rmd for the main file.

[otoomet]

I'd add a third option here: c) use the first file specified in the config file to look for the yaml header, and do not require the input argument at all.

[cderv]

input in render_book() is also there for the preview feature. Usually the behavior is:

• render_book() on a folder to render the whole book
• render_book() on a file different than input.Rmd for preview chapter when preview = TRUE

So input must be kept.

a) let the user choose an arbitrary file name

The input argument of render_book() is not used to indicate which is the file that contains the YAML header.

b) state clearly that index.Rmd is needed (and capitalization matters).

I thought it could be relaxed but it seems we have always assumed that index.Rmd was the file to use. We may have been to loose before.

For example, this is the file that is supposed to contain site: bookdown::bookdown_site() for Build site feature in RStudio IDE

bookdown/R/site.R

Lines 86 to 92 in 4ac16f0

	find_book_proj = function(input) {
	# if bookdown_site() is executed it is because site: has been set in index.Rmd
	rules = matrix(c(
	'^index.Rmd$', '^\\s*site:\\s*["\']?bookdown::bookdown_site["\']?\\s*(?:#.*)?$'
	), ncol = 2, byrow = TRUE, dimnames = list(NULL, c('file', 'pattern')))
	xfun::proj_root(input, rules)
	}

This is also what serve_book() is expecting

bookdown/R/utils.R

Line 391 in 4ac16f0

rebuild('index.Rmd', preview_ = FALSE) # build the whole book initially

bookdown/R/utils.R

Lines 394 to 400 in 7e9b46f

	first_html_format = function() {
	fallback = 'bookdown::gitbook'
	if (!file.exists('index.Rmd')) return(fallback)
	formats = rmarkdown::all_output_formats('index.Rmd')
	formats = grep('gitbook|html|bs4_book', formats, value = TRUE)
	if (length(formats) == 0) fallback else formats[1]
	}

c) use the first file specified in the config file to look for the yaml header

rmd_files config is not always passed so we can't only rely on that only. As said above, there is several feature of bookdown that expect a index.Rmd. Using the first file of rmd_files would be a big change for those feature.

Clearly, we are not coherent in all part of the code base regarding if index.Rmd is needed. I recognized that this was loose constraint, and enforcing it now is breaking some workflow that were working (almost because without index.Rmd I am not sure all bookdown project feature I working).

I'll need to reconsider this for next version. Thanks for reporting, I'll think about this more.

[otoomet]

Thanks for being so responsive!

[grimbough]

I'd like to add my 2 cents that we've been maintaining the book Modern Statistics for Modern Biology for several years now where the front page is generated from index.rmd provided to the input argument. As far as I'm concerned bookdown had been working perfectly in that usecase (if some features are missing I haven't noticed them).

my reading from the docs is that index.Rmd is just the default starting point for directories, but if one submits another file name then it is not needed. This is how I understood the docs, and up to now it also worked in this way:

My interpretation of the documentation was the same as @otoomet that index.Rmd was the default but replaceable. I'm certainly surprised at the case sensitivity here - capitalised file extensions feel weird and I normally expect .R Vs .r etc to be treated the same. Perhaps that distinction can be dropped, even if the general index filename requirement stands?

Thanks for maintaining this super useful package.

[cderv]

Thanks for your feedback @grimbough !

What will do based on all your feedbacks:

• We will support both extensions, index.rmd and index.Rmd - reverting to previous behavior and improving feature that was suppose to work only with index.Rmd

• We will modify the fix from use index.Rmd file to help detect output format when none provided #1332 introduced to fix v0.26 throws Error "argument is of length zero" if the Rmd file is in the sub-directory #1331 so that index.Rmd is not required but correctly used if found, and with a fallback mechanism if not found.

[cderv]

@otoomet @grimbough #1351 can be tested to see if this fix things for you. If you want to try it out.

remotes::install_github("rstudio/bookdown#1351")

[otoomet]

Thanks, now works for me even if I remove the dummy index.Rmd (there is index.rmd).

I thought that someone simply had overlooked an old hard-coded value, but it seems bookdown is quite a complicated project.

[cderv]

Thanks for confirming. And yes bookdown is quite a complicated workflow, also it is an now old project and it is always a non-so-easy task to adapt.

.Rmd remains the usual extensions expected by many project, and index.Rmd the main file where metadata are in project. Hopefully, the PR will help with likish file.

Thanks for opening the issue.

[github-actions]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.