Empty mediabag can't be remove when project is on another drives on Windows

[cderv]

Found while investigating #4593

Take this simple example

---
title: "test"
format: html 
---

Test

I put that on another drives than when Quarto is installed and called example K:

It will error

quarto render test.qmd
pandoc
  to: html
  output-file: test.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: test

ERROR: NotFound: Le fichier spécifié est introuvable. (os error 2), readdir 'test_files\mediabag'

NotFound: Le fichier spécifié est introuvable. (os error 2), readdir 'test_files\mediabag'
    at Object.readDirSync (deno:runtime/js/30_fs.js:120:16)
    at removeIfEmptyDir (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/path.ts:49:31)
    at renderCleanup (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/cleanup.ts:103:3)
    at file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render.ts:293:9
    at withTiming (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/core/timing.ts:37:20)
    at Object.complete (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render.ts:292:7)
    at async Object.onPostProcess (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-files.ts:565:28)
    at async renderFiles (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-files.ts:521:7)
    at async render (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/render-shared.ts:101:18)
    at async Command.fn (file:///C:/Users/chris/Documents/DEV_R/quarto-cli/src/command/render/cmd.ts:210:26)

First investigation shows that Deno.readDirSync has an issue on Windows reading this mediabag dir

• Folder is there on disk, with no content

> tree
Structure du dossier pour le volume KBFS
Le numéro de série du volume est 0000-0000
K:.
└───test_files
    ├───mediabag
    └───libs
        ├───bootstrap
        ├───clipboard
        └───quarto-html
> ls test_files/mediabag

but from Deno REPL

// folder is seen
> Deno.lstatSync("test_files/mediabag")
{
  isFile: false,
  isDirectory: true,
  isSymlink: false,
  size: 4,
  mtime: 2023-03-03T09:38:54.289Z,
  atime: 2023-03-03T09:38:54.289Z,
  birthtime: 2023-03-03T09:38:54.289Z,
  dev: null,
  ino: null,
  mode: null,
  nlink: null,
  uid: null,
  gid: null,
  rdev: null,
  blksize: null,
  blocks: null
}

// folders are listed
> for (const _entry of Deno.readDirSync("test_files")) {
  console.log(_entry)
}
{ name: "libs", isFile: false, isDirectory: true, isSymlink: false }
{ name: "mediabag", isFile: false, isDirectory: true, isSymlink: false }
undefined

// but Deno will error on reading mediabag
> for (const _entry of Deno.readDirSync("test_files/mediabag")) {
  console.log(_entry)
}
Uncaught NotFound: Le fichier spécifié est introuvable. (os error 2), readdir 'test_files/mediabag'
    at Object.readDirSync (internal:runtime/js/30_fs.js:119:14)
    at <anonymous>:2:27

We currently rely on this to remove empty dir (existsSync return TRUE for mediabag)

quarto-cli/src/core/path.ts

Lines 47 to 52 in 86b22ca

	if (existsSync(dir)) {
	let empty = true;
	for (const _entry of Deno.readDirSync(dir)) {
	empty = false;
	break;
	}

Looking at the code for Deno.emptyDir which is supposed also to check for emptyness, it seems the TryCatch on NotFound error (that we have here) is important

https://github.com/denoland/deno_std/blob/635da0f56c8c7b2294b0448763bda1d055dcfe44/fs/empty_dir.ts#L11

This is also the function we use to create the mediabag dir , so it seems the right way

quarto-cli/src/command/render/render.ts

Lines 97 to 98 in dee0d81

	const mediabagDir = filesDirMediabagDir(context.target.source);
	ensureDirSync(join(dirname(context.target.source), mediabagDir));

So something is not ok here.

IMPORTANT This does not happen when I am in the same drive on Windows as quarto - mediabag is correctly remove for the same qmd file.

[cderv]

So it seems this is indeed a Deno issue somehow

Test file

// what ensureDirSync is doing when folder does not exist
Deno.mkdirSync("emptyFolder", { recursive: true })

// Folder is there
Deno.lstatSync("emptyFolder")

// Let's try to read into it
for (const _entry of Deno.readDirSync("emptyFolder")) {
  console.log(_entry)
}

• Run this on Windows inside a working dir on another drive letter than the Quarto install

> quarto run test.ts
error: Uncaught NotFound: Le fichier spécifié est introuvable. (os error 2), readdir 'emptyFolder'
for (const _entry of Deno.readDirSync("emptyFolder")) {
                          ^
    at Object.readDirSync (deno:runtime/js/30_fs.js:120:16)
    at file:///K:/public/cderv/test.ts:8:27

>  rm emptyFolder

• Run this now on folder in drive where quarto is and it works without error

> quarto run test.ts
> rm emptyFolder

Same with Deno REPL

[cderv]

It is quite specific and not encountered in #4593 so does not seem that urgent. I probably need to dig deeper to understand if this can be a hint to potential more issues regarding Quarto and multiple drives usage on Windows

[jefferds]

I have the same problem using google drive. Error with google drive also occurred at #2174. In the quarto-1.2.475-win version these problems do not occur.

[lisalevinson]

I'm also having this problem after updating to 1.3 with files on a Google Drive path in Windows.

[cderv]

Similar issue

• Error with removing intermediary files after updating to 1.3 #5430

But this time it happens also before removing empty dir and for intermediates *_files

---
title: test
embed-resources: true
---

```{r}
plot(1 + 1)
```

this leads to

ERROR: NotFound: Le fichier spécifié est introuvable. (os error 2), remove 'G:/Mon Drive/Projects/test-quarto/test_files'

NotFound: Le fichier spécifié est introuvable. (os error 2), remove 'G:/Mon Drive/Projects/test-quarto/test_files'
    at Object.removeSync (deno:runtime/js/30_fs.js:157:9)
    at file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:80560:22
    at Array.forEach (<anonymous>)
    at renderCleanup (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:80558:39)
    at file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:81106:46
    at withTiming (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:19624:25)
    at Object.complete (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:81106:13)
    at async Object.onPostProcess (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:86673:36)
    at async renderFiles (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:86644:13)
    at async render (file:///C:/Users/chris/scoop/apps/quarto/current/bin/quarto.js:90675:21)

It seems to be a Deno issue with network drive. If I enter the project in terminal and run Deno, here is what I get

> import { existsSync } from "https://deno.land/std@0.185.0/fs/exists.ts";
undefined
> existsSync('G:/Mon Drive/Projects/test-quarto/test_files')
true
> Deno.removeSync('G:/Mon Drive/Projects/test-quarto/test_files', {recursive: true})
Uncaught NotFound: Le fichier spécifié est introuvable. (os error 2): remove 'G:/Mon Drive/Projects/test-quarto/test_files'
    at Object.removeSync (ext:deno_fs/30_fs.js:175:7)
    at <anonymous>:2:6

So even if the folder exists and seen as existing by Deno, removeSync seems to not find it - possibly an issue in Deno ?
We'll see - denoland/deno#18996

[jefferds]

I made the following comment to bridge the problem in 'pathto/Quarto/bin/quarto.js', it's working, I think!:

function removeIfEmptyDir(dir) {
    if (existsSync(dir)) {
        let empty = true;
        for (const _entry of Deno.readDirSync(dir)){
            empty = false;
            break;
        }
        // if (empty) {
        //     Deno.removeSync(dir, {
        //         recursive: true
        //     });
        //     return true;
        // }
        return false;
    } else {
        return false;
    }
}

[py9mrg]

Same issue for me with latest Quarto version when using folder in G Drive. Switching file to .Rmd solves the problem.

[cderv]

Switching file to .Rmd solves the problem.

Switching to .Rmd extension on the file and using still Quarto solved the issue ? Or switching to not using Quarto and using rmarkdown::render() directly ?

[py9mrg]

I switched the file extension to rmd and then clicked "Open Preview ..." in VS Code, so whichever that does! I think probably the latter because IIRC .rmd in VS Code is handled by the vscode-r extension, not quarto?

[cderv]

Yes probaby the later. It ran rmarkdown directly without quarto, so indeed you don't have this Quarto specific issue.

[py9mrg]

I mean when I used a .qmd extension it was definitely using quarto and I got this same issue as listed in the original post and subsequent discussions. When I changed the filename to .rmd I think it switched to standard rmarkdown rendering, which is why it then worked.

[jgutman]

I believe this is the cause of this issue: rstudio/rstudio#13813, originally reported here: rstudio/rstudio#11804 (comment)

[neellab-umd]

Did this issue of Quarto not rending when the files are on Google drive ever get addressed? It is happening in the "Desert Sunflower" Release (cd7011dc, 2023-10-16) for windows, 2023.09.1 Build 494 and Quarto v 1.3.

Or is there a workaround besides recoding everything in RMarkdown? I start teaching Quarto in my R class on Monday and if rendering is not working with google drive I will have a different weekend ahead of me than I thought.

[cscheid]

I start teaching Quarto in my R class on Monday and if rendering is not working with google drive I will have a different weekend ahead of me than I thought.

I'm sorry about your weekend :/ But this issue has existed for over a year, and we don't think it's a fix we can easily do ourselves, unfortunately (we believe it's coming from the underlying filesystem implementation interacting badly with Deno, the programming language runtime we use.)

[neellab-umd]

Thanks for the quick response. It has been a year since I taught my class so I missed this.

I saw a comment above from March 3 that this bug was specific and did not seem urgent. It sounds like windows users who work on remote drives make up a small part of the Quarto user base. Unfortunately for me, I and almost all of my students fit this demographic.

Is there a timeframe for this bug being fixed? It sounds like the resolution is outside of RStudio. Given that, is it going to be a permanent feature that on Windows Quarto scripts can't be rendered on a shared drive? I don't mean to sound snarky or like I think the universe revolves around me. I appreciate all the hard work that goes into this ecosystem. I am really just trying to get a sense of the future. Last year when Quarto was announced it seemed like that was where things were headed. I jumped on the bandwagon so I was not teaching my students old ways. If the development and support of Quarto has changed, it seems like it might prudent to go back to teaching Rmarkdown.

Thanks for any info you can provide so I can make a decision about what to teach.

[cderv]

This is still an issue with Deno 1.33.3 and std@0.209.0/fs/exists.ts

Moving to next release as I don't think we can fix in Quarto without Deno fix.

[neellab-umd]

I was just about to circle back to this issue. Would it be best for me to post an issue to the Deno group? I might be able to articulate the problem in acceptable way for them, but I am not sure.

FWIW - I went ahead and used Quarto with my class this semester. But this issue made most of them (n=26) less that enthusiastic about using Quarto for their research. Most of them work on their thesis or dissertation research with Google Drive mirrored on their computers so that their files are backed up to the cloud automagically. I also try to instill behaviors that prevent proliferation of similar or related files in different places. I had to go against both of those good practices and create a parallel project for Quarto that they were to keep on their hard drives. Most of them hated this dual project cludge with no cloud backup and some recoded the qmd's back to rmds so they did not have to deal with it. Despite demonstrating all the great things they can do with Quarto, the requirement to do all work on the hard drive did not leave them wowed by the tool.

I know my class is a small drop in the bucket for you, but I feel sad to not have 26 new Quarto converts. Last year when it worked normally almost everyone LOVED it.

Let me know if posting an issue to the Deno github page is the way to go.

Thanks,
Maile Neel

[cderv]

Would it be best for me to post an issue to the Deno group? I might be able to articulate the problem in acceptable way for them, but I am not sure.

I opened one already

• removeSync() NotFound error while folder exists when used on Network Drive on Windows denoland/deno#18996

Most of them work on their thesis or dissertation research with Google Drive mirrored on their computers so that their files are backed up to the cloud automagically.

Just ideas: they would benefit to learn about Git. Quarto project can easily be tracked with Git, and any remote repository tool (Github, Gitlab, Gitbucket, ...)

Working on code project from Synced folder is not easy. I always add bad experience with IDE and any other tool that don't like the real time syncing.

I wonder if pausing sync while working, and syncing only at the end of the day helps with this issue... 🤔

[neellab-umd]

HA! I KNEW you were going to suggest Git. I have think of it every year, but a) most of my students have never seen R or any command line before and it is enough to cover R and RStudio in the semester. But I will revisit that between now and next fall when I teach the class again.

FWIW, RStudio plays really nicely with a synced GoogleDrive. I have been using it for years and only once have had any issues with sharing a drive among collaborators. It does not give you the control over revisions that Github does, but for my work it is great - and the learning curve is about 0.

Pausing syncing does nothing to fix the rendering "error" issue with Quarto. Believe me, I tried many "creative" workarounds. On the bright side, you can actually render, you just can't see it in the RStudio Viewer and you have to look past a slew of warnings. But the product is there.

[cscheid]

FWIW, RStudio plays really nicely with a synced GoogleDrive

That's great to hear but not anything that we can control. Quarto and RStudio are entirely different codebases; we can't "take the code from RStudio" or anything like that, unfortunately.

[cderv]

Considering the new behavior I found (detailed at denoland/deno#18996 (comment)), I believe I have a fix for this specific issue with mediabag.

[cderv]

@neellab-umd the specific issue with mediabag should be fixed in next Quarto 1.4 build.

However, we cannot yet ensure that Quarto is working well on Remote drives like Google Drive. Hopefully, this could be better behavior for you right now, but you may encounter other limitation and side effects.

[neellab-umd]

That is great news! Can you say when the next build will be released? I am looking forward to testing it out. Hopefully it fixes this issue without unintended consequences.

Thank you so much!

[cderv]

Next one should be v1.4.529 - it should be built later today or tomorrow. You can watch out for this version, or I'll ping you here

[neellab-umd]

Hi Christophe -

I have tested v1.4.31 and my .qmd scripts on Google Drive now render perfectly to html. They throw no errors and the html shows up in the viewer as expected. Thank you so much!

Unfortunately, pdfs and docx still throw errors. They do render, but the errors are still showing. I have pasted the error messages below. Do you need more information about my setup or does this give you what you need?

Rendering PDF
running xelatex - 1
This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode

running xelatex - 2
This is XeTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023) (preloaded format=xelatex)
restricted \write18 enabled.
entering extended mode

ERROR: NotFound: The system cannot find the file specified. (os error 2): remove 'X:/My Drive/1_1_2023_R-Class/Course.Project.2023/code/Unit.10.B.practice.makes.purrr.fect_files'

Stack trace:
at Object.removeSync (ext:deno_fs/30_fs.js:189:7)
at file:///C:/PROGRA1/Quarto/bin/quarto.js:69298:22
at Array.forEach ()
at renderCleanup (file:///C:/PROGRA1/Quarto/bin/quarto.js:69296:39)
at file:///C:/PROGRA1/Quarto/bin/quarto.js:70062:50
at withTiming (file:///C:/PROGRA1/Quarto/bin/quarto.js:19429:25)
at Object.complete (file:///C:/PROGRA1/Quarto/bin/quarto.js:70062:17)
at eventLoopTick (ext:core/01_core.js:183:11)
at async Object.onPostProcess (file:///C:/PROGRA1/Quarto/bin/quarto.js:77146:36)
at async renderFileInternal (file:///C:/PROGRA~1/Quarto/bin/quarto.js:77130:5)

[cderv]

X:/My Drive/1_1_2023_R-Class/Course.Project.2023/code/Unit.10.B.practice.makes.purrr.fect_files

Do you see this path still exists after the error is thrown ?

It could be another occurrence of the problem in the codebase.

[neellab-umd]

I am not sure what you are asking me to check. That path to my qmd still exists on my system - but I am not sure how to tell if quarto sees it exists.

[cderv]

I meant the path of the error

X:/My Drive/1_1_2023_R-Class/Course.Project.2023/code/Unit.10.B.practice.makes.purrr.fect_files

So the *_files folder that goes with your .qmd. Quarto is trying to remove it, and if this is the same bug, I believe it succeeded by still throws an error (because of explanation given in this thread)

[neellab-umd]

Sorry to be dense! Yes - that path still remains. So Quarto is not succeeding in removing the *_files folder.

[cderv]

it could be a different issue then. I'll look into it.

[neellab-umd]

Thank you!!

[py9mrg]

Hello all,

For me (using version 1.4.532) I still get the error, but not on mediabag, I get it on filename_files?

[cderv]

@py9mrg see latest discussions above and #7993