Path problems with system calls on Windows #6
Comments
|
Hi,
Thanks for filing this bug report and sorry to hear you’re having trouble getting the package to work. I’m pretty sure this is actually related to a bug that I already fixed in the latest version of the package posted on GitHub, but not yet on CRAN. If you are running the CRAN version, it would be great if you could download the latest development version from GitHub and let me know if it fixes your issue. (You will need to restart your R session after reinstalling). If this doesn’t work, please let me know as I definitely want to get this sorted before submitting the next version to CRAN, which will hopefully be coming soon. Thanks!
Update: I could have sworn I fixed this leading slash issue before, but now it's seeming like that may not be the case. I just pushed a patch to the main branch that I think should solve this issue (070f297). It would be great if you can test it for me
…On Mon, Nov 21, 2022 at 9:59 PM Phenomniverse ***@***.***> wrote:
Hi,
I'm struggling to get read_chroms to accept a path_out, and its unclear
what formatting it is looking for. It seems to want to add a "/" to the
front of whatever I specify as path_out. The default behaviour is supposed
to save it in the current working directory, but it doesn't do that either,
it prompts whether to save it in 'temp', but it can't find 'temp' folder
even when I manually created it as a subfolder of the working directory. I
would prefer to give the full path from the drive name (eg "C:/ ... "), but
this doesn't work because its putting a "/" in front of the drive name. See
below where I've used getwd() to provide the path to the current directory.
read_chroms(paste0(archive_sample_dirs[1],'/FID1A.ch'),find_files=FALSE, path_out=getwd(), export=TRUE, parser="openchrom", format_in='csd',export_format = 'csv')
Error in read_chroms(paste0(archive_sample_dirs[1], "/FID1A.ch"), find_files = FALSE, :
The export directory '/W:/ARL/Analytical/OPERATOR METHOD TEMPLATES/chemstation data/' does not exist.
If I make path_out = "", it ignores this problem at least for long enough
to encounter an additional problem, which is that it can't find the path to
the OpenChrom command line. It seems to want the pathname with filename,
sans extension, eg "C:/Users/ ... /Programs/OpenChrom/openchrom" for this.
If I type this in, it moves on to another error (I suspect its back to the
first error). It seems to save this path in the
path_to_openchrom_commandline.txt file, but it doesn't seem to be able to
find OpenChrom unless I manually type it in each time. When I do so, I get:
Error in write_xml.xml_document(x, file = path_xml) : Error closing file
In addition: Warning message:
In write_xml.xml_document(x, file = path_xml) : Permission denie [1501]
—
Reply to this email directly, view it on GitHub
<#6>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADZEBO7ITD4LQ7OROKGFXZTWJQZJXANCNFSM6AAAAAASHJQZOU>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
|
Regarding the OpenChrom path, I think on Windows you need the full path to the |
|
Thanks Ethan I'll try it out tomorrow when I'm back in my office. Would you mind giving me an overview of the command line arguments that OpenChrom requires to convert a csd file to csv? I couldn't find any OpenChrom documentation on this point. |
|
What you have looks good for converting just one file. Theoretically in the latest version you shouldn't need the You may need to run |
|
Oh you meant for OpenChrom, sorry. You need to provide an xml batch file to the OpenChrom command line to access the parsers. Then you run |
|
(Maybe I should split the batch file constructor code off into a separate function in case people want to call that separately to just make the batch files). I think that might make for cleaner code anyway |
|
Also, If you'd be open to sharing one of those FID files with me that would be helpful for testing the package. I don't think I have any FID files from Agilent in my little collection yet |
I reinstalled the package from github but the path_out issue still occurs. The error message no longer includes the path_out value, so I can't confirm whether this is because of the leading slash still being present or if there is another issue. Not specifying a path_out results in being prompted to accept export to 'temp' directory, which also doesn't work:
I attempted this (I ran the configure_rainbow() function first, but I don't think it was necessary, I think that ran automatically when I loaded the package). This is the result:
Sounds like a good idea to me.
I should be able to find a chemstation data file that is okay to share. What's the best way to get it to you? |
|
Ahh sorry this still isn't working. Thank you for the detailed report!
I'm pretty sure I know why this wasn't working and just patched it. Fixing the path issue on windows is going to be a bit trickier. I'll need to spend some time troubleshooting on the Windows box in the lab which I won't be able to do until next week because of the holiday. I clearly introduced a bug somewhere but I can't quite figure out what the problem is. The back slashes in the windows paths make a lot of problems in R... If you wanna to just email me an example file (at ethanbass@gmail.com) that would be great. I can confirm for you if you want that |
|
So I actually messed around with the paths a little more just now and pushed another update to fix the way the paths are parsed on Windows. I have the OpenChrom parser working again on my Windows 10 computer in the lab. Might be worth another look when you get a chance. I also confirmed that the rainbow parser seems to be running smoothly now at least on my installation of Windows 10. |
|
Ok so this is a bit weird but the path_out option seems to work for the rainbow parser, but not the openchrom one. However, the rainbow parser is returning null rather than collecting the data from the .ch file. But it is successfully creating the empty .csv file in the working directory. So that's progress. Another issue that I forsee when I get the parser to work properly is that the output .csv file is named as per the input .ch file, but if read_chroms is iterating over a list of input files that all have the same name (in different directories), I'll only end up with a .csv of the last .ch file evaluated. Chemstation saves its raw data as FID1A.ch by default and I have a lot (thousands) of these files so its not ideal to be changing the input file names. |
Email sent :) |
|
Thank you for the update and for sending along the files. I'm glad to hear that the rainbow parser is running for you now (even though it isn't actually reading your files). Unfortunately, I checked out your files and it doesn't seem like any of the parsers that are currently included with the package are able to read them properly. I know Roderick Bovee who develops the entab package is working on an update for the agilent FID parser (bovee/entab#42) and it seems like the rainbow developers might also be interested in your file (since they have a parser for agilent FID files that is throwing an error on your file). Do you know what version of Chemstation your files are created by? With your permission, I'd be happy to pass your files along to Roderick and the rainbow people and they might be able to update the parsers to properly read your files. Regarding the path issue with OpenChrom, I'm pretty perplexed and don't really understand why this is happening since it's working fine on the other Windows computer I have access to. I will have to look into this further. Regarding the file name issue, if you have the files in the original Thank you for all the feedback and please let me know if it would be OK to share the files. |
|
It seems that your file is from B.04 chemstation and maybe the rainbow parser can only handle files from b.03 chemstation |
|
I'll confirm the chemstation version when I'm back in the office tomorrow. It's possible we have different versions on different machines so I might be able to try more than one version. I will email you re: sharing FID files. |
|
OK sounds good. I am making some progress by the way figuring out the binary. I don't usually do this myself but I decided to try and I am getting back a chromatogram that looks a lot like the one you sent me, so I'm pretty sure it shouldn't be too hard to write a parser directly or R or adapt the one in the rainbow package |
|
I added a new parser in ba59590 to read these agilent FID files natively in R. It seems to work for both the 179 type files and 181 type files. If anyone reading this has a 180 type file to see where it falls along this spectrum that would be helpful. |
|
Ok, our GCs are mostly running various iterations of Chemstation B.04.03, but we do have a couple on OpenLab CDS Chemstation Edition, and a couple on a newer version of OpenLab. We also have an GCMS running MSD Chemstation D.02.00.275. We have GCFID data going back to 2013, and maybe earlier, which I suspect has been collected using older versions of chemstation. Your parser (parser="chromconverter", format_in='chemstation_fid') works pretty well for more recent files, it doesn't work so well with the 2013 file that I tried. With the newer files, the actual chromatogram data is good, which is the important bit, but some of the attribute fields that read_chroms generates are a bit off. For my purposes this information is more readily accessed through the other files stored in the .D folder anyway (eg .xls, .xml, .txt files, depending on what was specified in the chemstation acquisition method). Regarding the older files (tested on one from 2013 and another from 2014), the parser isn't working as expected. The rt (x-axis) data seems okay, but the value (y-axis) data is very strange. See image below (I've ploted as points because the line plot just fills the whole screen with black). Also, the attributes fields contain a lot of unreadable characters. I'm not sure what version of chemstation these older files were generated by, although maybe its contained in the software field, if it could be interpreted. The newer file just had "Asterix ChemStation" in that field though.
|
|
Further to the above, the chromconverter parser works well for FID data generated on OpenLab CDS Chemstation Edition C.01. |
|
Thanks for sending the additional files and doing all this testing. I actually have a function in the package already that reads the chemstation XLS files and attaches some of the metadata. I forgot to "activate it" for the new parser, but probably that would be worth doing. I will try to fix up the attribute fields from the binary files as well when I have time, thanks for the tips. I'm not sure what's going on with the other chemstation files you sent yet. They seem to be encoded differently than the newer files even though they're also the "181"-type files |
ethanbass
changed the title
|
@Phenomniverse I just updated the agilent FID parser to a new version that can read the older 181 files you sent me. (06a80bf). Give it a whirl and please let me know if you have any other FID files that aren't being translated correctly. |
|
Hi @ethanbass happy new year! Great to know that you're still working on this! Ok, so I tested the latest update against about a dozen random GCFID files dating back to ~2004, and it appears to be working nicely, I haven't found any that it doesn't work for yet, but will keep you updated if I do. I did have to tweak my code a little bit because it appears that the structure of the output from read_chroms has changed a little bit with this latest update. Previously I was able to convert the read_chroms output to a dataframe and it would have two columns capturing the x- and y-axis values respectively. Now it seems that the x-axis data is appearing as the row names when converted to a data frame. This isn't a big problem, I can change it back to the format that the rest of my code expects with one line. Just curious as to why the change in the structure of the read_chroms output? |
|
Glad it seems to be working. Sorry for the unexpected change re: the format of the data.frames. This is still not completely consistent between different parsers wrapped by the |
|
No its all good. The output read_chroms generates has the advantage of including the meta data from the FID file, which I lose as soon as I turn it into a data frame with two columns only. It's only for the sake of my existing code that I am converting it into a data frame. Thanks again for your efforts on this project! |
|
Thank you as well for all your helpful feedback! By the way, the data.frame generated by |
|
read_chroms outputs as a list: I turn it into a dataframe using: The setDT function from data.table package is converting the rownames into a column here. Maybe a better way to do it would be: I suppose I could pass the attributes from fid$FID1A.CH to fid_df as a comment, but there's no real need for it. |
|
You could do something like this if you want to transfer the metadata over: |
|
yeah okay, playing around with this makes me realise that retaining the metadata is pretty helpful. |
|
Yes, I actually noticed that too when I was working on the parser.
Unfortunately, the metadata is encoded a bit differently in the older
files. I have to work on some other things for a while, but I will try to
revisit this at some point and try to come up with a solution
…On Sun, Jan 15, 2023 at 10:07 PM Phenomniverse ***@***.***> wrote:
yeah okay, playing around with this makes me realise that retaining the
metadata is pretty helpful.
I notice that the metadata for some of those older FID files contains some
strings of nonsense characters although the actual metadata is in the
strings as well. For example, see below, where some of the attribute fields
are normal and others contain extraneous characters :
$ attributes:List of 11
..$ version : chr "181"
..$ sample_name: chr "REF TTOØ\022\024\002\002‘|E"
..$ run_date : chr "12/12/2018 12:03:24 AM"
..$ instrument : chr "HP G1530AÀ&$÷j‘|\030ß\022"
..$ method : chr "REF-TTO.MÐý\177àý\177Ðý\177:4@\004x(e$Hß\022"
..$ software : chr ***@***.***\002\a$á\022\a$‘|\177'‘|\002\a´à\022\034á\022Dá\022\235'‘|X ***@***.***\002\a$á\022n ***@***.***\0"| __truncated__
..$ unit : chr "pA"
..$ signal : chr ""
..$ time_range : num [1:2] 1.82e-04 3.00e+01
..$ data_format: chr "long"
..$ parser : chr "chromConverter"
—
Reply to this email directly, view it on GitHub
<#6 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADZEBOZTZBRJQLAZRIXPYRLWSS3OFANCNFSM6AAAAAASHJQZOU>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
@Phenomniverse FIY this issue with the metadata in the chemstation 181 files should be resolved in the latest version (v0.4.0). Also, you can now toggle the format of the data.frames using the |
Hi,
I'm struggling to get read_chroms to accept a path_out, and its unclear what formatting it is looking for. It seems to want to add a "/" to the front of whatever I specify as path_out. The default behaviour is supposed to save it in the current working directory, but it doesn't do that either, it prompts whether to save it in 'temp', but it can't find 'temp' folder even when I manually created it as a subfolder of the working directory. I would prefer to give the full path from the drive name (eg "C:/ ... "), but this doesn't work because its putting a "/" in front of the drive name. See below where I've used getwd() to provide the path to the current directory.
If I make path_out = "", it ignores this problem at least for long enough to encounter an additional problem, which is that it can't find the path to the OpenChrom command line. It seems to want the pathname with filename, sans extension, eg "C:/Users/ ... /Programs/OpenChrom/openchrom" for this. If I type this in, it moves on to another error (I suspect its back to the first error). It seems to save this path in the path_to_openchrom_commandline.txt file, but it doesn't seem to be able to find OpenChrom unless I manually type it in each time. When I do so, I get:
The text was updated successfully, but these errors were encountered: