How to open a file inside <py-script>?

[LakshmanKishore]

Opening a file with open(filename) function throws FileNotFound Error.

app/
 - index.html
 - abc.txt
 - app.css

Inside the index.html file if we try to open the abc.txt file using <py-script> open("abc.txt","r") </py-script> throws a FileNotFound Error.

[Rjtsahu]

You cannot read/open a file from a local disc, as the code is running in the browser and would not have access to the file system, but you can definitely download the given file over the network (using the native fetch function)

[LakshmanKishore]

@Rjtsahu So you are saying that we cannot access the abc.txt file inside the <py-script></py-script> tags?

[ckavidas]

@LakshmanKishore

That is exactly what he is saying, your py-script code is running in web assembly container which is separate from the files on disk.

You will need to upload/download files depending on your use case, see #111 and #124

[Shahin-rmz]

@LakshmanKishore

That is exactly what he is saying, your py-script code is running in web assembly container which is separate from the files on disk.

You will need to upload/download files depending on your use case, see #111 and #124

Good to know, new to front end.

[LakshmanKishore]

@ckavidas Thanks for the description.
If the file needs to be opened in a binary mode like open('model.pkl','rb') then open_url(url) does not provide the option of opening file with binary mode.
open('model.pkl','rb') opens a saved ML model that was dumped using pickle module.
Is there a way to open that pickle file within <py-script></pyscript>?

[Rjtsahu]

@LakshmanKishore you need to load the file from http/fetch

from js import fetch
response = await fetch(URL)

please share the snippet of what you are trying to achieve, so I can tell exactly what you can do.

[LakshmanKishore]

I have a KNN ML model that was built with scikit-learn and dumped to iris.pkl file using pickle module.

<py-script>
import pickle
from pyodide.http import open_url
url = "https://raw.githubusercontent.com/LakshmanKishore/irisClassification/master/iris.pkl"
model = pickle.load(open_url(url))
#sl-sepal length pw-petal width   I will get this from the html form element.
data = {"sl":Element("sl").value, "sw":Element("sw").value, "pl":Element("pl").value, "pw":Element("pw").value }
def predict(*ags, **kws):
    prediction = model.predict([[float(data["sl"]),float(data["sw"]),float(data["pl"]),float(data["pw"])]])
    classes = ["SETOSA","VERSICOLOR","VIRGINICA"]
    answer = classes[int(prediction)]
    print(answer)
</py-script>

I tried the above snippet but it was showing

JsException(PythonError: Traceback (most recent call last): File "/lib/python3.10/site-packages/_pyodide/_base.py", line 429, in eval_code .run(globals, locals) File "/lib/python3.10/site-packages/_pyodide/_base.py", line 300, in run coroutine = eval(self.code, globals, locals) File "", line 4, in TypeError: a bytes-like object is required, not 'str' )

Since pickle.load(file.pkl) needs the file.pkl to be opened in binary mode, whereas open_url opens the file in str format.
Once the pkl file is loaded, I will use model.predict() to get the predictions of the new input data.

[ckavidas]

@LakshmanKishore

See this example

In the code you can see upload file/read file operations which include converting to raw bytes.

[LakshmanKishore]

import pickle

class StrToBytes:
    def __init__(self, fileobj):
        self.fileobj = fileobj
    def read(self, size):
        return self.fileobj.read(size).encode()
    def readline(self, size=-1):
        return self.fileobj.readline(size).encode()

from pyodide.http import open_url
url = "https://raw.githubusercontent.com/LakshmanKishore/irisClassification/master/iris.pkl"
model = pickle.load(StrToBytes(open_url(url)))

I was able to convert the model into bytes format using the above StrToBytes class, but showing error when loading the model.

JsException(PythonError: Traceback (most recent call last): File "/lib/python3.10/site-packages/_pyodide/_base.py", line 429, in eval_code .run(globals, locals) File "/lib/python3.10/site-packages/_pyodide/_base.py", line 300, in run coroutine = eval(self.code, globals, locals) File "", line 13, in _pickle.UnpicklingError: invalid load key, '\xef'. )

I don't know whether this error is due to pkl file or the usage of the pickle inside <py-script> tag.

[Rjtsahu]

@LakshmanKishore can you verify if it is working in normal mode (python3 standard version), you can try to read the file directly using open.
Let's make sure it's not a pickle file issue.

[LakshmanKishore]

Pickle file is working perfectly fine when the file is located and accessed locally, but when I tried to get the pickle file from online(like from the raw.githubusercontent.com it is showing line 13, in _pickle.UnpicklingError: invalid load key, '\xef'. error
I'm guessing that the pickle file may be getting corrupted when we try to load the file using the url.

[marimeireles]

Hey @LakshmanKishore, I'm not super knowledgeable with pickle stuff but it seems like this is no longer related to pyodide. A quick google search returned stuff like: leftthomas/ESPCN#6.
Can you confirm this or the problem persists?
Thanks!

[LakshmanKishore]

Hello @marimeireles
I updated the pickle file.
I'm still getting the same UnpicklingError.

[marimeireles]

Closing it as dup. please redirect any questions related to the topic to the documentation issue.
If you're facing a different problem or think might have encountered a bug please feel free to open a new issue.
If I misinterpreted your issue and you think it should still be open please reply here and we'll reopen it.
Thank you!

[LakshmanKishore]

I was able to open the file using

[[fetch]]
-->url here

inside the pyscript.toml file,
which will fetch the file in the envirnoment, so that it can be loaded in the python code.

I have implemented in project1 and project2

[JeffersGlass]

Excellent! For those finding their way here, starting in 2023.11.1.RC3 there is also the files feature of py-config, which has a simpler interface.