New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rdata, read R datasets from Python #144
Comments
Hi @vnmabus, just letting you know we have started the search for an editor for this review. In the meantime, here are the initial editor checks. I am happy to report that rdata passes with flying colors. Please see a couple of comments below. You are not required to address these for us to start the review, but I do think doing so would benefit your package--just trying to help, you get a free review from the editor in chief too 🙂 Editor in Chief checksHi there! Thank you for submitting your package for pyOpenSci Please check our Python packaging guide for more information on the elements
Editor commentsTwo suggestions:
More on adding data: >>> parsed = rdata.parser.parse_file(rdata.TESTDATA_PATH / "test_vector.rda")
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/pimienta/Documents/repos/coding/opensci/pyos/test-rdata/.venv/lib/python3.10/site-packages/rdata/parser/_parser.py", line 1002, in parse_file
data = path.read_bytes()
File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/pathlib.py", line 1126, in read_bytes
with self.open(mode='rb') as f:
File "/home/linuxbrew/.linuxbrew/opt/python@3.10/lib/python3.10/pathlib.py", line 1119, in open
return self._accessor.open(self, mode, buffering, encoding, errors,
FileNotFoundError: [Errno 2] No such file or directory: '/home/pimienta/Documents/repos/coding/opensci/pyos/test-rdata/.venv/lib/python3.10/site-packages/rdata/tests/data/test_vector.rda' I think what's going on is that the test data is not in the built package? You may have intended this snippet to be run with just the development version, but I would strongly suggest you add small amounts of data to the package itself. It's also possible you meant to "include" the data in your built package, but the setuptools build is not configured correctly? I have found the way setuptools does this a bit confusing in the past, but I think it's now possible for you to include all the files without using a MANIFEST.in file (which if I understand correctly is now a "legacy" way of including files with setuptools that should no longer be used). Note that other build backends have their own methods for this; for example flit would include everything by default, although you can have more fine grained control. You can then provide access to this data to users (and yourself, for tests) by using importlib-resources as discussed in this talk. Here's an example of using importlib-resources in a package I develop--it might be a little easier to read than more general scientific Python packages that have a lot of internal infrastructure around their built-in datasets. I have a draft section of the guide on how to do this here--it's very rough still but the core ideas are there and might help you. If you have any feedback on this section, we'd love to hear it. |
@NickleDave I have added the test data to the manifest and changed the package to use Some observations of the process:
|
Great, thank you @vnmabus--this feedback is really helpful. I will read this in more detail, and share in our Slack team, but I want to let you know right away I appreciate it. It might be worth sharing your observations on the process you went through to include data in our Discourse, if you wouldn't mind starting a new topic there, since that's open and we can get more eyeballs on it: https://pyopensci.discourse.group/. We could share the topic on social media too (e.g. Mastodon). I just sent out an email to another potential editor today--will update you as soon as we have one! We want to make sure we find someone with the right expertise in both Python and R. |
or if you're ok with it @vnmabus I can start the topic on our Discourse and you could feel free to reply there. Please let me know We'd like to get input from maintainers of setuptools there, to better understand when we need MANIFEST.IN files -- this has come up before in GitHub review of our packaging guide |
If you can start the topic and give me a link, that would be awesome, thank you! |
Hi again @vnmabus I asked a question about MANIFEST.in here, please feel free to chime in if I'm not understanding all the complexities of your particular situation: |
Hi @vnmabus, I'm very happy to report that @isabelizimm will be the editor for this review. We are now looking for reviewers. |
@NickleDave And I am also happy to report that I have added a couple of more detailed examples, as per your second suggestion. Please, tell me if that is what you had in mind: https://rdata.readthedocs.io/en/latest/auto_examples/index.html |
These look great @vnmabus, thanks so much. This is exactly what I had in mind. |
Hey there! I'm excited to be the editor for this package--I can't count the amount of times I've done the R->Python data dance. I'm on the hunt for reviewers, and will check back end of next week with an update! |
👋 Hi @rich-iannone and @has2k1! Thank you SO MUCH for volunteering to review for pyOpenSci 🎉 @vnmabus these two reviewers are individuals who are deeply involved in both the R and Python data and packaging world. I am super excited to be part of and learn from them through this review process! Please fill out our pre-review surveyBefore beginning your review, please fill out our pre-review survey. This helps us improve all aspects of our review and better understand our community. No personal data will be shared from this survey - it will only be used in an aggregated format by our Executive Director to improve our processes and programs.
The following resources will help you complete your review:
If anyone has any questions/comments/concerns, please don't hesitate to get in touch! Your review is due: Dec 8 |
Hi @vnmabus, First, I am happy to review rdata. Second, currently there is no tag for 0.9.2.dev1. Though, as there aren't many changes between when this issue was opened (after v0.9.1) and today, I am fine with addressing my review to any version/commit after that. |
Yes, sorry, the 0.9.2.dev1 version was the number of the develop version when I opened this issue. As I added some breaking changes, it was renamed and published until version 0.10 instead. That said, it would be better if you include in your review also the recent changes in develop (I know it is a moving target, but this package does not usually change that much). In particular, there is a recent PR from a user that greatly improves XDR parsing speed, removing the |
That is okay with me. On |
Package ReviewPlease check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
DocumentationThe package includes all the following forms of documentation:
Readme file requirements
The README should include, from top to bottom:
NOTE: If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the a badge for pyOpenSci peer-review will be provided upon acceptance.)
UsabilityReviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole.
Functionality
For packages also submitting to JOSS
Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted. The package contains a
Final approval (post-review)
Estimated hours spent reviewing: 3 Review CommentsThe software is good and useful. I find myself having a need for this very thing. In terms of letting people know what the package does, it's somewhat terse at the Other use cases can be presented to get the casual user's imagination going, as well. This may involve hosting .rds files (maybe in a public GH repo) and demonstrating Some additional documentation things:
Package README and Documentation Site The organization of the project's GH page is pretty good. Without a direct link, however, it's a bit hard to search for (appearing on the 4th page of results) and that's A useful badge to have is the Python versions supported badge. The list of versions could match those that you're testing and the relevant Community Considerations To make things better for community contributions, I suggest adding these two things:
You already have a CONTRIBUTING document that describes ways to make contributions and this is great. The PR and issue templates can link to that document (this increases I recommend adding a CODE_OF_CONDUCT.md file. These are very commonplace so you can get a file from another project's repo. Code Comments I found the code to well organized and easy to follow. All conventions seem to be followed and many helpful docstrings were present for virtually all classes and methods. Testing It's good to see that testing is being done on a solid matrix of recent Python versions and platforms (Linux/Ubuntu, Windows, MacOS). There does need to be some updates Another recommendation is to run pylint for linting, perhaps through pre-commit (see https://pylint.pycqa.org/en/latest/user_guide/installation/pre-commit-integration.html). There are a lot of tests in place and very good line coverage. Having the coverage results sent to CodeCov makes it very easy to see which code paths go through tests. It's Perhaps consider adding Python 3.12 to the testing matrix. If you want to go even further with testing, have different R versions on different platforms generate .rds/.rda files and test those directly. Citation File The citation file should have a version field, updated every time a version of the software is released. This can be a bit of a pain but it makes for a better, more |
Package ReviewPlease check off boxes as applicable, and elaborate in comments below. Your review is not limited to these topics, as described in the reviewer guide
DocumentationThe package includes all the following forms of documentation:
Readme file requirements
The README should include, from top to bottom:
NOTE: If the README has many more badges, you might want to consider using a table for badges: see this example. Such a table should be more wide than high. (Note that the a badge for pyOpenSci peer-review will be provided upon acceptance.)
UsabilityReviewers are encouraged to submit suggestions (or pull requests) that will improve the usability of the package as a whole.
Functionality
For packages also submitting to JOSS
Note: Be sure to check this carefully, as JOSS's submission requirements and scope differ from pyOpenSci's in terms of what types of packages are accepted. The package contains a
Final approval (post-review)
Estimated hours spent reviewing: 5 hours Review CommentsNotes & Suggestions
parsed = rdata.parser.parse_file("dataframe.rds")
data = rdata.conversion.convert(parsed) would be data = rdata.read_rds("dataframe.rds") And maybe also a These functions could include optional arguments to pass on to the parser and/or converter. This would simplify the user experience as the
Final CommentWhile I have noted some suggestions, the project is otherwise well motivated, implemented and tested and I do not consider the above suggestions to be "blockers". |
THANK YOU!!!! To @has2k1 and @rich-iannone for your very thorough reviews 🙌 your time is so appreciated, and I think a great asset to this project!
I have gone ahead and changed the status of the issue to awaiting changes. There were a number of updates suggested by the reviewers; some of these comments will be quick to implement, and others might take a bit more time. This part of the review is a bit more back and forth, as @vnmabus updates Thank you all again for your collaboration! |
I will be on Christmas holiday from 14th December to 14th January, and will likely not have access to my computer. I will try to address as many issues as possible today and tomorrow, but it is likely that I won't be able to address them all until January. I hope that does not cause you many inconveniences. |
No inconvenience at all, please unplug and enjoy your Christmas holiday! I'm adding an on-hold tag on this review to signify to everyone that no further work is expected until mid-January upon your return. When you're back and ready to pick up changes to |
So I am back, and I think that I implemented most of the requested changes (in the develop branch, not yet released). First, for @rich-iannone:
Missing badges have been added
I am not sure if you reviewed the latest version, because I think the links were in the examples documentation page before your review. Please, tell me if that is not enough.
Again, I am not sure if you reviewed the latest version. I made changes to both the README and the main page of the documentation after submitting but before your review. Please tell me if something can be improved.
We do not have performance claims (although with the recent changes to the XDR parser, performance has been greatly improved).
I added Ruff to the CI and fixed all violations.
I plan to write it after the review, if possible, to accommodate all the changes.
As mentioned before, I am not sure if you reviewed the latest changes because a set of examples was added in response to @NickleDave comments. It is true that the examples do not illustrate a complete workflow, just the loading part. Do you consider that is really necessary?
I added asv tests. Currently I only test the array parsing routines, as their performance has greatly improved recently, but if you have additional proposals for performance tests I will consider adding them.
I created a page in the documentation for this. Please, tell me if this is what you had in mind.
I am not sure if I follow you here. Could you please clarify your intention?
I added the link (thanks!) and removed the "Packages" section. I am unsure of which additional search tags I could use.
Added.
I added both of them.
I linked to it in the PR template.
That was already added before your review IIRC.
Done.
I have finally chosen Ruff for its performance. I added it to CI in order to report violations (but I do not want them to be automatically fixed without human intervention, so it just checks them).
Added.
I want to improve testing, and I had several ideas. I created an issue for tracking that.
I am a bit unsure about it. Adding a version field has the risk that the user does a copy-paste of the generated Bibtex from the Github repo, which may not be the same version that he is using. If I do not add it, either the user cites the package or he retrieves the version from his copy if he is interested in reproducibility. In both cases, the information should not be wrong. Is there a compelling argument for having the version there? |
Now, for @has2k1:
Added.
I just noted that the link is not in the README, added now.
I do not have it yet. I should probably include a comparison at least with pyreadr, but I am not sure how to frame it.
There is a CITATION.cff file. I plan to add citation to the README after publishing in JOSS (if they approve it).
I made no performance claims.
I have tests, typing and style as Github actions.
I added Ruff as a linter and fixed all violations.
I plan to add it after the review by PyOpenSci is complete.
Done.
These convenience functions have been incorporated.
I added a page in the documentation. Please tell me if that is what you had in mind.
Everything moved to
The PR has been merged and |
Ah, this is a great round of improvements-- thank you for the update @vnmabus! For @has2k1 and @rich-iannone, are you able to comment on if these fixes are what you expected? |
@vnmabus has resolved the all core queries and addressed the 2 conditional issues. To keep track of one, I have filed an issue about the comparison with other python packages, and the In between, the tables for default conversions are great and super informative. Great work Carlos. |
Sorry for being very late here, but thank you @vnmabus for all the improvements made. I've made a few comments in error previously so I thank you for the clarifications on those! The examples are great, no need to go further beyond the loading part (which is the key thing here). As far as adding a version number to the citation file, I take what I said back since I'm in favor of your reasoning (and you bring up some good points about practical usage of the citation text). Testing is always a potential marathon, you're doing great with this and the inclusion of asv tests is very helpful. The page in the documentation about the supported R translations is quite useful. Thanks for adding that in. To wrap up, this package fully meets my quality and usability expectations. Excellent work! |
With two reviewers giving the 👍 ... it is my absolute pleasure to say that ... 🎉 ... Thank you @vnmabus for submitting Author Wrap Up TasksThere are a few things left to do to wrap up this submission:
It looks like you would like to submit this package to JOSS. Here are the next steps:
🎉 Congratulations! You are now published with both JOSS and pyOpenSci! 🎉 Editor Final ChecksPlease complete the final steps to wrap up this review. Editor, please do the following:
If you have any feedback for us about the review process please feel free to share it here. We are always looking to improve our process and documentation in the peer-review-guide. |
Thank you @isabelizimm for the smooth reviewing experience, and also thanks to the reviewers @rich-iannone and @has2k1! I already did the steps at section "There are a few things left to do to wrap up this submission:". I will plan to do the necessary steps to submit to JOSS and create a blog post in the next weeks. |
@isabelizimm I wanted to tell you that I have written a draft of a blog post, and I am not sure how to send it to you for review. On an unrelated note: rdata shows now in the main pyOpenSci page, but it is not shown in a particular category (maybe it should be shown in "data munging"?). |
hey @vnmabus just a quick note that i saw this message and noticed a bug in our website! things should be working and rdata should filter as expected now: |
So @isabelizimm , @lwasser : is it possible to add a blog post or not? |
Apologies for late response-- you are welcome to make a PR with your post to the pyOpenSci website's repo! Here is an example of what this PR can look like 😄 pyOpenSci/pyopensci.github.io#42 |
Submitting Author: Name (@vnmabus)
All current maintainers: (@vnmabus)
Package Name: rdata
One-Line Description of Package: Read R datasets from Python.
Repository Link: https://github.com/vnmabus/rdata
Version submitted: 0.9.2.dev1
Editor: @isabelizimm
Reviewer 1: @rich-iannone
Reviewer 2: @has2k1
Archive: TBD
JOSS DOI: TBD
Version accepted: 0.11.0
Date accepted (month/day/year): 2/29/2024
Code of Conduct & Commitment to Maintain Package
Description
.rda
and.rds
files, containing serialized R objects, and convert them to Python. The users can influence this conversion and provide conversion routines for custom classes.Scope
Please indicate which category or categories.
Check out our package scope page to learn more about our
scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):
Domain Specific & Community Partnerships
Community Partnerships
If your package is associated with an
existing community please check below:
Its main purpose is to be able to read .rda and .rds files, the files used for storing data in the R programming language, and convert them to Python objects for further processing.
The target audience includes users that want to open in Python datasets created in R. These include scientists working in both Python and R, scientists who want to compare results among the two languages using the same data, or simply Python scientists that want to be able to use the numerous datasets available in CRAN, the R repository of packages.
The package rpy2 can be used to interact with R from Python. This includes the ability to load data in the RData format, and to convert these data to equivalent Python objects. Although this is arguably the best package to achieve interaction between both languages, it has many disadvantages if one wants to use it just to load RData datasets. In the first place, the package requires an R installation, as it relies in launching an R interpreter and communicating with it. Secondly, launching R just to load data is inefficient, both in time and memory. Finally, this package inherits the GPL license from the R language, which is not compatible with most Python packages, typically released under more permissive licenses.
The recent package pyreadr also provides functionality to read some R datasets. It relies in the C library librdata in order to perform the parsing of the RData format. This adds an additional dependency from C building tools, and requires that the package is compiled for all the desired operating systems. Moreover, this package is limited by the functionalities available in librdata, which at the moment of writing does not include the parsing of common objects such as R lists and S4 objects. The license can also be a problem, as it is part of the GPL family and does not allow commercial use.
@tag
the editor you contacted:#143
Technical checks
For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:
Publication Options
JOSS Checks
paper.md
matching JOSS's requirements with a high-level description in the package root or ininst/
.Note: JOSS accepts our review as theirs. You will NOT need to go through another full review. JOSS will only review your paper.md file. Be sure to link to this pyOpenSci issue when a JOSS issue is opened for your package. Also be sure to tell the JOSS editor that this is a pyOpenSci reviewed package once you reach this step.
Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?
This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.
Confirm each of the following by checking the box.
Please fill out our survey
submission and improve our peer review process. We will also ask our reviewers
and editors to fill this out.
P.S. Have feedback/comments about our review process? Leave a comment here
Editor and Review Templates
The editor template can be found here.
The review template can be found here.
Footnotes
Please fill out a pre-submission inquiry before submitting a data visualization package. ↩
The text was updated successfully, but these errors were encountered: