Export reports

[mcarmack]

Hello-

This is my first pull request to this project- I hugely appreciate any feedback or improvements you may have!

I added a method to wrap the redcap API "export reports" method.

Three things I am looking to learn/incorporate into my code before this pull request is considered:

1. I saw in other methods there is some handling of the format "df" - I assume dataframe? As far as I can see in the RedCap API documentation, the formats data is returned in will only be csv, json, xml, or odm (cdisc odm xml, odm version 1.3.1). So is the handling of the format 'df' unnecessary within this method? I see it is used in 'export_records' pycap method (which has the same format/return format options as export reports, per RedCap API documentation), so I included it in my export_reports method. Please advise.

2. It is unclear to me from the RedCap API documentation exactly what is returned from the API call. Apparently it is, "Data from the project in the format and type specified ordered by the record (primary key of project) and then by event id". This leads me to believe it will be a list of records from a single report. If so, I'm not sure why the method is called "export reports" instead of "export report". Also, will all fields be returned, or just those of the report?

3. Testing! I do not understand how to test this method without having it be part of the version of pycap I am using. Also, I would like to write some tests for the test file, but I would also appreciate guidance on best practice for that.

Thank you again for your help!

[sburns]

Hi @mcarmack, thanks for this Pull-Request! I'm trying to stay away from the computer this holiday break :) I'll be able to give more feedback next week.

[sburns]

Sorry for the delay on this! Overall this looks great, though to be honest I haven't truly studied PyCap in a long time :/ With respect to your specific questions...

1. To me, pandas is a huge deal and is a large part of what makes the Python data ecosystem so great. REDCap's API has no idea what pandas is though so in various export_ methods you'll us doing things like if format == 'df': ret_format = 'csv. This means we really want to ask the REDCap API for data in CSV format. Then after the API requests finishes, we transform that blob of CSV to a DataFrame and return it to the user. PyCap just makes this really easy. Technically speaking, csv also tends to be the smallest payload across the wire because it includes headers only once and not per record like XML or JSON.

2. I too don't know what you're gonna get from the API! Have you been able to run this locally? Reports were a thing when I was still working with data on REDCap, but they were early. But generally they're tabular output of some fields with rows based on some filters, right? So I would assume that you'll get that format from the API.

3. This is a great question. We're using responses so that we don't need an actual REDCap instance running when tests run. But I didn't add this so I'd really have to dig to see what's going on. In general, before most of the tests run, we're adding registering callbacks into responses that will try to match certain URLs/request bodies. Then, when the tests run and PyCap tries to make actual HTTP requests with the requests package, responses jumps in and returns these fake responses. Because we don't exactly know what the API response looks right now for a report, any test we right wouldn't be super useful (yes we could make the test pass, but the code still might not be correct in practice, if that makes sense.

Have you been able to run export_reports locally?

[sburns]

hi @mcarmack have you been able to look at this anymore?

[mcarmack]

@sburns - hello and thank you so much for the comments! This all makes sense now. I ended up going right to the redcap API to do export_reports (without the pycap wrapper).

Export reports does return a list of records, as if you had done export_records with all the record ids in that report

Appreciate all your thoughts!

[pwildenhain]

This seems good to merge for people who want this functionality, I just pushed a change to resolve conflicts. If you still want to merge @mcarmack, let me know. And if you have any hesitation @sburns also, let me know

[xd3262nd]

@mcarmack Hi, do you mind to share how you export reports via API and getting the project info?

[pwildenhain]

Exporting project info: https://pycap.readthedocs.io/en/latest/api.html#redcap.project.Project.export_project_info

import redcap
url = "your redcap api url"
token  "your project token"

project = redcap.Project(url, token)
project.export_project_info()

Once this PR is merged, you'll be able to export reports like so:

project.export_reports(report_id="your report id")

[xd3262nd]

@pwildenhain I have attached the screenshoot below, there isn't any export_project_info() in the projects.py file I downloaded

[pwildenhain]

Weird, because it's in the current master branch: https://github.com/redcap-tools/PyCap/blob/master/redcap/project.py#L671. Maybe trying pulling, or re-cloning?

[xd3262nd]

@pwildenhain I am sorry but how can I do the re-cloning? and call it? As I am using google collab and not sure how will that works

[pwildenhain]

Ah, I'm not sure how dependency management works with google collab 🤷‍♂️