Metadata not readable

[defra1]

Hi, I am trying out ethoscopy to analyse my ethoscope data on my windows 11 Pro.

When I load my metadata.csv file (which I have been able to use with the Rethomics analysis software in R), it says that it is not readable.
I am using docker desktop for windows.

See the code below:

etho.__version__
'1.2.9'

import ethoscopy as etho import pandas as pd

meta_loc = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data 2/metadata.csv'
remote = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data 2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/'

the data is saved on my computer after being downloaded from the node

local = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data 2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/'

meta = etho.link_meta_index(meta_loc, remote, local)

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[49], line 1
----> 1 meta = etho.link_meta_index(meta_loc, remote, local)

File /usr/local/lib/python3.10/dist-packages/ethoscopy/load.py:239, in link_meta_index(metadata, remote_dir, local_dir)
    237         print("An error occurred: ", e)
    238 else:
--> 239     raise FileNotFoundError("The metadata is not readable")
    241 if len(meta_df[meta_df.isna().any(axis=1)]) >= 1:
    242     print(meta_df[meta_df.isna().any(axis=1)])

FileNotFoundError: The metadata is not readable

This is the format of my metadata.csv file:

<style> </style>

machine_name	date	region_id	condition
ETHOSCOPE_001	2022-08-23	1	HPD
ETHOSCOPE_001	2022-08-23	2	HPD
ETHOSCOPE_001	2022-08-23	3	HPD
ETHOSCOPE_001	2022-08-23	4	HPD
ETHOSCOPE_001	2022-08-23	5	HPD
ETHOSCOPE_001	2022-08-23	6	HPD
ETHOSCOPE_001	2022-08-23	7	HPD
ETHOSCOPE_001	2022-08-23	8	HPD
ETHOSCOPE_001	2022-08-23	9	HPD
ETHOSCOPE_001	2022-08-23	10	HPD
ETHOSCOPE_001	2022-08-23	11	HSD
ETHOSCOPE_001	2022-08-23	12	HSD
ETHOSCOPE_001	2022-08-23	13	HSD
ETHOSCOPE_001	2022-08-23	14	HSD
ETHOSCOPE_001	2022-08-23	15	HSD
ETHOSCOPE_001	2022-08-23	16	HSD
ETHOSCOPE_001	2022-08-23	17	HSD
ETHOSCOPE_001	2022-08-23	18	HSD
ETHOSCOPE_001	2022-08-23	19	HSD
ETHOSCOPE_001	2022-08-23	20	HSD

It is saved as a UTF-8(Comma delimited) (*.csv). Thank you

[ggilestro]

I have no experience with docker on windows but at first sight this does not look right:

meta_loc = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data 2/metadata.csv'

Aren't you supposed to mount your files to a local location inside the docker?
When you run the docker container you probably have to mount your volume with a command like this one:

-v /c/Users/franc:/home/franc

Then from ethoscopy you should be able to access the file in

/home/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data 2/metadata.csv

[defra1]

@ggilestro Yes, when I initiated the container I followed the instructions on: https://bookstack.lab.gilest.ro/books/ethoscopy/page/getting-started for windows.

sudo docker run -d -p 8080:8080 \
      --name ethoscope-lab \
      --volume /z/ethoscope_data:/mnt/ethoscope_results:ro \
      --restart=unless-stopped \
      ggilestro/ethoscope-lab

I have modified it because the port 8080 did not work at first.

docker run -d -p 8000:8000
 --name ethoscope-lab\
--volume --volume /c/Users/franc/Desktop:/mnt/home/franc \
--restart=unless-stopped  ggilestro/ethoscope-lab \

When I open up the container to look at the files I can see my Desktop items mounted into the container:

And when I look a the Bind Mount I can see my desktop mounted as well

In the Jupyter Notebook I execute the same commands after loading
meta_loc = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data_2/metadata.csv'
remote = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data_2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/'
local = '/c/Users/franc/Desktop/Ethoscope_results/ethoscope_data/ethoscope_data_2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/'

After trying

meta = etho.link_meta_index(meta_loc, remote, local)

I still get the same error message.

[ggilestro]

You're correctly mounting /c/Users/franc/Desktop to /mnt/home/franc which is where you need to point your script to. All your scripts should therefore read:

meta_loc = "/mnt/home/franc/Ethoscope_results/ethoscope_data/ethoscope_data_2/metadata.csv"

[defra1]

@ggilestro The new path seemed to solved the metadata issue. However, I still cannot create the meta code, since I am not using an ftp server to transfer my .db files from the node to a separate computer for analysis, I do not have an ftp account.
If I keep the same three elements

meta_loc = '/mnt/home/franc/Ethoscope_results/ethoscope_data/ethoscope_data_2/metadata.csv' 

remote = '/mnt/home/franc/Ethoscope_results/ethoscope_data/ethoscope_data_2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/'

local = 
'/mnt/home/franc/Ethoscope_results/ethoscope_data/ethoscope_data_2/results/00190f0080e54d9c906f304a8222fa8c/ETHOSCOPE_001/2022-08-23_03-33-59/' 

The remote option just has the same path as local (not sure if that is correct) because the .db files are on my local computer.
When I run
meta = etho.link_meta_index(meta_loc, remote, local)

I get this error

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
Cell In[39], line 1
----> 1 meta = etho.link_meta_index(meta_loc, remote, local)

File /usr/local/lib/python3.10/dist-packages/ethoscopy/load.py:278, in link_meta_index(metadata, remote_dir, local_dir)
    276 parse = urlparse(remote_dir)
    277 ftp = ftplib.FTP(parse.netloc)
--> 278 ftp.login()
    279 ftp.cwd(parse.path)
    280 files = ftp.nlst()

File /usr/lib/python3.10/ftplib.py:412, in FTP.login(self, user, passwd, acct)
    403 if user == 'anonymous' and passwd in {'', '-'}:
    404     # If there is no anonymous ftp password specified
    405     # then we'll just use anonymous@
   (...)
    409     # - We don't want to let ftp sites to discriminate by the user,
    410     #   host or country.
    411     passwd = passwd + 'anonymous@'
--> 412 resp = self.sendcmd('USER ' + user)
    413 if resp[0] == '3':
    414     resp = self.sendcmd('PASS ' + passwd)

File /usr/lib/python3.10/ftplib.py:280, in FTP.sendcmd(self, cmd)
    278 def sendcmd(self, cmd):
    279     '''Send a command and return the response.'''
--> 280     self.putcmd(cmd)
    281     return self.getresp()

File /usr/lib/python3.10/ftplib.py:207, in FTP.putcmd(self, line)
    205 def putcmd(self, line):
    206     if self.debugging: print('*cmd*', self.sanitize(line))
--> 207     self.putline(line)

File /usr/lib/python3.10/ftplib.py:202, in FTP.putline(self, line)
    200 if self.debugging > 1:
    201     print('*put*', self.sanitize(line))
--> 202 self.sock.sendall(line.encode(self.encoding))

AttributeError: 'NoneType' object has no attribute 'sendall'

If I run the next code without including remote
meta = etho.link_meta_index(meta_loc, local)

I get this message

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
Cell In[40], line 1
----> 1 meta = etho.link_meta_index(meta_loc,local)

TypeError: link_meta_index() missing 1 required positional argument: 'local_dir'

[ggilestro]

@Lblackhurst29 can you please update the documentation page ( https://bookstack.lab.gilest.ro/books/ethoscopy/page/loading-the-data ) to include the syntax to be used when loading the files directly from the machine that holds the dbs? The doc only point to the remote option. Thanks.

[Lblackhurst29]

Hi @defra1, this was actually an oversight from when I changed things from R and so there was no way to create a linked metadata file without an FTP server, which was silly. Given at this point even if you have it saved to a FTP sever you will have downloaded it to your local machine of have it mounted it isn't needed at all.

So i've re-written that part of the code (hopefully making it quicker too) to remove the remote_dir variable, it just searches you local path for the db files. One note, for the path just give the details for the directory that contains all the data, not an individual folder. So it should be '/mnt/home/franc/Ethoscope_results/ethoscope_data/ethoscope_data_2/results/' for you. The function searches all the sub directories, so it has the potential to break.

If you could use the newest version - 1.2.24 - and see if that works you. It looks to working within our systems (all linux), however I don't have a windows machine to test it on. But I re-wrote it with windows in mind. Just call etho.link_meta_index(meta_loc, local) now

The docs should also be updated to reflect the changes.

[defra1]

@Lblackhurst29 Thank you for upgrading the documentation. When I open up a new notebook with the python3 kernal and type in
pip install ethoscopy
It doesn't look like it can find the new version

Defaulting to user installation because normal site-packages is not writeable
Requirement already satisfied: ethoscopy in /usr/local/lib/python3.10/dist-packages (1.2.9)
Requirement already satisfied: hmmlearn<0.3.0,>=0.2.8 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (0.2.8)
Requirement already satisfied: tabulate<0.9.0,>=0.8.10 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (0.8.10)
Requirement already satisfied: numpy<2.0.0,>=1.22.3 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (1.24.2)
Requirement already satisfied: PyWavelets<2.0.0,>=1.4.1 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (1.4.1)
Requirement already satisfied: colour<0.2.0,>=0.1.5 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (0.1.5)
Requirement already satisfied: astropy<6.0.0,>=5.1.1 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (5.2.2)
Requirement already satisfied: scipy<2.0.0,>=1.8.0 in /usr/lib/python3/dist-packages (from ethoscopy) (1.8.0)
Requirement already satisfied: kaleido==0.2.1 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (0.2.1)
Requirement already satisfied: plotly<6.0.0,>=5.7.0 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (5.14.0)
Requirement already satisfied: pandas<2.0.0,>=1.4.2 in /usr/local/lib/python3.10/dist-packages (from ethoscopy) (1.5.3)
Requirement already satisfied: pyerfa>=2.0 in /usr/local/lib/python3.10/dist-packages (from astropy<6.0.0,>=5.1.1->ethoscopy) (2.0.0.3)
Requirement already satisfied: packaging>=19.0 in /usr/local/lib/python3.10/dist-packages (from astropy<6.0.0,>=5.1.1->ethoscopy) (23.0)
Requirement already satisfied: PyYAML>=3.13 in /usr/local/lib/python3.10/dist-packages (from astropy<6.0.0,>=5.1.1->ethoscopy) (6.0)
Requirement already satisfied: scikit-learn>=0.16 in /usr/local/lib/python3.10/dist-packages (from hmmlearn<0.3.0,>=0.2.8->ethoscopy) (1.2.2)
Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3/dist-packages (from pandas<2.0.0,>=1.4.2->ethoscopy) (2022.1)
Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas<2.0.0,>=1.4.2->ethoscopy) (2.8.2)
Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.10/dist-packages (from plotly<6.0.0,>=5.7.0->ethoscopy) (8.2.2)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.1->pandas<2.0.0,>=1.4.2->ethoscopy) (1.16.0)
Requirement already satisfied: joblib>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.16->hmmlearn<0.3.0,>=0.2.8->ethoscopy) (1.2.0)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn>=0.16->hmmlearn<0.3.0,>=0.2.8->ethoscopy) (3.1.0)

[notice] A new release of pip available: 22.3.1 -> 23.3
[notice] To update, run: python3 -m pip install --upgrade pip
Note: you may need to restart the kernel to use updated packages.

There is probably a simple solution to this, I have upgraded pip, and restarted the kernal, but none of this has worked to get the newer version. Would you have any other suggestions? Perhaps this is a windows issue? I do not have access to a linux computer unfortunately.

[defra1]

@Lblackhurst29 Forget the last comment, I was able to find the latest version and install it and now I don't have any errors with the meta command.