Improve compatibility with windows (e.g. can't init)

[riziles]

Describe the bug

context
When I run myst init, I get an error message:

$ myst init
SyntaxError: Invalid regular expression: /c:\LocalTemp\Repos\Local\MystTest\/: \ at end of pattern

problem
I realize this project is still in beta, but it looks awesome. I'm a big JupyterBook user and love this idea. Maybe my bug is specific to Windows?

Reproduce the bug

run myst init from Windows command line. Same error in Powershell and Git Bash as well. Also tried manually typing in the path.

List your environment

node 18.12.1
Windows 10

[welcome]

Thanks for opening your first issue here! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out EBP's Code of Conduct. Also, please try to follow the issue template as it helps other community members to contribute more effectively.

If your issue is a feature request, others may react to it, to raise its prominence (see Feature Voting).

Welcome to the EBP community! 🎉

[rowanc1]

So glad you are jumping in. :)

Can you try running with myst init -d for debug mode?

Most of our testing is on mac and Linux so would love your help getting up and started with some more windows usage!

[riziles]

Here is the output of myst init -d:

SyntaxError: Invalid regular expression: /C:\LocalTemp\Repos\Local\MystTest\/: \ at end of pattern
    at new RegExp (<anonymous>)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216265:25
    at Array.map (<anonymous>)
    at replaceCwd (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216263:15)
    at Object.info (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216280:50)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:224651:19
    at Generator.next (<anonymous>)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:78:61
    at new Promise (<anonymous>)
    at __async (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:62:10)
    at init (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:224608:10)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:225052:13
    at Generator.next (<anonymous>)
    at fulfilled (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:65:24)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)


C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216265
      return a2.replace(new RegExp(cwd + import_path8.sep, "g"), "");
                        ^

SyntaxError: Invalid regular expression: /C:\LocalTemp\Repos\Local\MystTest\/: \ at end of pattern
    at new RegExp (<anonymous>)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216265:25
    at Array.map (<anonymous>)
    at replaceCwd (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216263:15)
    at Object.error (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:216290:50)
    at C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:225061:19
    at Generator.throw (<anonymous>)
    at rejected (C:\LocalTemp\Miniconda\node_modules\myst-cli\dist\myst.js:72:29)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)

[rowanc1]

We have tracked down that specific error -- path replacement for nicer logging was using a regular expression, which isn't a great idea in retrospect. That has been replaced! I am going to rename this issue to capture more compatibility issues with windows. We will do a bit more testing and then do a patch release early next week -- hopefully you can try it out then!

[rowanc1]

@riziles the specific issue that you ran into has been released as myst-cli 0.1.4

@stevejpurves was going to do a bit more exploratory testing on Windows later today and might run into some similar issues if you want to wain a few more days. :)

Leaving this open until then.

[riziles]

Seems to work now. Ran into an issue trying to export PDF that might be related to Windows vs POSIX command line conventions.

[stevejpurves]

some notes from testing on windows with 0.1.4:

• myst init is working
• myst start works for 64 bit versions of nodejs only, as the turbo build tool does not support 32bit windows architecture
• myst build article.md --tex appear to complete successfully
• myst build article.md --pdf fails during pdf creation with an error "the syntax of the command is incorrect". the command in question is:

latexmk -f -xelatex -synctex=1 -interaction=batchmode -file-line-error
-latexoption="-shell-escape" data-driven-latex-templates.tex &> data-driven-latex-templates.shell.log

the issue is the pipe to the log file. The equivalent syntax for windows systems is:

latexmk -f -xelatex -synctex=1 -interaction=batchmode -file-line-error 
-latexoption="-shell-escape" data-driven-latex-templates.tex 1> data-driven-latex-templates.shell.log 2>&1

which will pipe stdout and stderr to the file which we should use when process.platform === 'win32'

• the pdf generated by the command above cannot be opened and the pdf reader thinks it is corrupt.
  I used this file from my blog for testing
  Looking at the logs:

1. the framed environment is not available. there are various uses of it in my test file, adding \usepackage{framed} to the preamble in my generated tex file fixes this - so perhaps an issue with the imports function/algo?
2. all of the images are not found, each with an error like this:

Latexmk: Missing input file 'images040aVXqR51yjBy-a444210719e70b2f4892f220b4c52010.png' (or dependence on it) from following:
  LaTeX Warning: File `images040aVXqR51yjBy-a444210719e70b2f4892f220b4c52010.png' not found on input line 189.

Note the full filepath on disk for this example in the error message is images\DOHMeg040aVXqR51yjBy-a444210719e70b2f4892f220b4c52010.png and that does exist in ful in the tex file.

\includegraphics[width=0.7\linewidth]{images\DOHMeg040aVXqR51yjBy-a444210719e70b2f4892f220b4c52010.png}

are we missing an escape on the backslash in the filepath?

@riziles any of that look familiar in terms of the export issues you were hitting

[stevejpurves]

Also as I've not reached the point of building a pdf successfully yet, I cannot be sure that the pdf will end up in the folder specified in exports.output, as I a using forward slashes in paths in my front matter, I'd also expect a problem there unless we're doing path conversion on that field.

[stevejpurves]

Also, I tried exporting a different, simpler article

myst build --pdf curvenote-principles.md
OR
myst build --pdf .\curvenote-principles.md

and hit the same error as expected, but given a clean _build/temp folder at the start I end up with 3 new subfolders -- two contain the tex output for my simpler article and the third contains the tex file and partial pdf build for the other article I was trying previously , which is the only other one that has exports frontmatter -- are command line arguments not getting picked up correctly on windows? as this is what we'd expect from running myst build --pdf with no filename

[rowanc1]

Ok, most things are working. Will follow up with another release soon!

[stevejpurves]

great! I realised that the issue with the temp folders I mentioned in the comment above was a red herring! (i had a file open and the folder wasn't being deleted) so the cli build command is sound and working as expected.

[riziles]

Sorry... didn't mean to hit the close button.... @stevejpurves , the myst build article.md --pdf error you saw is exactly what I was seeing.

[stevejpurves]

just tested v0.1.5 and the build problem is now resolved!
init, start and build working as expected

thanks for reporting the issue @riziles !

[riziles]

@rowanc1 , A few additional questions:

• How do I get the myst build --pdf to include Jupyter notebooks? Is that possible? I don't see any info in the docs on it.
• myst build --site appears to simply export some JSON files. Perhaps that is by design, but the GitHub README suggests that it is possible to export static HTML. Okay if not; just curious.
• Any chance we can get a search box on the documentation site?

[rowanc1]

You can build the notebooks, add an export field with the format and the template (https://github.com/myst-templates):

Note that the images (e.g. of dynamic figures) as well as code highlighting leaves some stuff to be desired. Should be improved a lot in the coming few weeks though! As well as some docs.

For the website, I hope that you have found the myst start?! It currently requires a web-app, to deploy the site easily, you can use a curvenote deploy (docs). We will work through some of the static templates that are just HTML/CSS files (probably more on the months timeline, issue). You can also self host, but more complicated. :)

I would love to do search, agree it is a big missing piece, see #100 if you have other thoughts there.

Please keep the requests coming (here or new issues), will get to them as fast as we can!!

[riziles]

Yes, myst start works great. I was asking about the html because my org has an internal doc site that only accepts static html or basic markdown. I personally use JupyterBook to generate my docs, but I love the live preview functionality offered by myst start. I can continue to use JupyterBook for the time being.

[rowanc1]

Awesome to know, I will aim to spend some time on the static HTML export next week. I know it is possible, just not quite sure how hard.

Thanks again for the feedback!