Fix No such file or directory: /tmp/mdkatex/...
#17
Conversation
|
I'm quite sure this change has the side effect of eliminating any caching between runs. If the issue is that files are being accessed concurrently, I think the more appropriate change would be to make sure all files are first written with a suffix |
|
I hadn't realized that caching was used in this way. I agree that this change breaks caching. Is the goal to enable caching within one process, or also cache between different invocations? |
|
Caching across multiple invocations / separate processes is the idea yes. |
|
Sorry for coming back late to this. In order to realize caching between multiple runs safely, one could use separate directories: (i) for storing cached results and (ii) for running |
- make file write operations atomic - use "cache" instead of "tmp" as that better reflects the nature of the files (whereas /tmp/ is only incidentally the directory we use
|
@tovrstra Is it possible for you to verify that my most recent changes fix the issue? |
|
Sorry for the delay. I've repeated the build process, which involves about 1200 Markdown to PDF conversions (feedback with equations for my students). These conversions run in parallel on a machine with 16 cores. With the latest version of this pull request, I get the following error: I cannot share this specific test case due to privacy concerns. (The traceback is also edited.) To facilitate fixing the issue, a simple example would be better. I'll see if I can cook up something that can be shared. |
|
Perfect, I appreciate the help. |
When multiple threads are running, they may both cleanup the cache directory at the same time and one thread will provoke FileNotFound errors in the other thread.
|
With my most recent commit, I'm reasonably sure, that your issue should be fixed. A test would of course be nice, regardless, but I'll take your word for it if you say everything is working now. |
|
Thanks for the additional commits! These do indeed fix the issue as far as my testing goes. I also created a Python script that reproduces the bug (before your latest commits). To see the error message, you need to remove all files under #!/usr/bin/env python
import concurrent.futures
import random
import markdown
TEXT_SNIPPETS = """
Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
""".split()
EQ_SNIPPETS = r"""
x y \sin(\theta) \hat{H} \vec{B} \int_0^\infty f(x)\,dx
\{i^2\}_{i=1}^N e^{2it\pi\omega} \tan(x)
""".split()
def generate_md() -> str:
parts = []
for _ in range(100):
parts.append(random.choice(TEXT_SNIPPETS))
parts.append(f"$`{random.choice(EQ_SNIPPETS)}`$")
return " ".join(parts)
def convert(md: str) -> str:
md_ctx = markdown.Markdown(
extensions=[
"fenced_code",
"markdown_katex",
"tables",
],
extension_configs={
"markdown_katex": {"insert_fonts_css": True}
},
)
return md_ctx.convert(md)
def main():
with concurrent.futures.ProcessPoolExecutor(max_workers=30) as executor:
mds = [generate_md() for _ in range(30)]
for html in executor.map(convert, mds):
print(html)
print()
if __name__ == '__main__':
main()It may be possible to simplify the script further and still get the error. |
|
@mbarkhau Would it be possible to make a release with this fix? With the current version of pip (24.1), markdown-katex cannot be installed, due to issues fixed in this PR. I get the following error when installing markdown-katex with the latest pip: Thanks! |
|
I just pushed v202406.1035 |
|
Thank you so much! I've tested it once more with the pip-installed version. It all works. |
This fixes issue #16 (So far, I've not run into this error anymore with this fix.)
I also had to correct the requirements file to make
pip install -e .work.See https://pip.pypa.io/en/stable/reference/requirement-specifiers/#examples