CTGOV changed website layout

[rfhb]

ClinicalTrials.Gov just made the previous beta website layout now the default layout. Links to the previous website start now with https://classic.clinicaltrials.gov/. Search parameters are different. ctrdata is being updated to handle the situation.

[rfhb]

Insert "classic." into search URLs, that is, change e.g.:

• from https://clinicaltrials.gov/ct2/results?cond=Neuroblastoma
• to https://classic.clinicaltrials.gov/ct2/results?cond=Neuroblastoma

Then try the development version with these install steps; it should work with such URLs and with previously stored queries:

install.packages("devtools")

devtools::install_github("rfhb/ctrdata", build_vignettes = TRUE)

library(ctrdata)

q <- "https://classic.clinicaltrials.gov/ct2/results?cond=Neuroblastoma&term=&intr=Investigational+Agent"

dbc <- nodbi::src_sqlite(collection = "my_collection")

ctrLoadQueryIntoDb(q, con = dbc)

# * Found search query from CTGOV: cond=Neuroblastoma&intr=Investigational+Agent
# Checking helper binaries: . . . done
# (1/3) Checking trials in CTGOV:
# Retrieved overview, records of 272 trial(s) are to be downloaded (estimate: 2.2 MB)
# Download status: 1 done; 0 in progress. Total size: 1.85 Mb (100%)... done!             
# (2/3) Converting to JSON, 272 records converted
# (3/3) Importing JSON records into database...
# = Imported or updated 272 trial(s)                 
# No history found in expected format.
# Updated history ("meta-info" in "my_collection")
# $n
# [1] 272
# 
# $success
#  [1] "NCT00002825" "NCT00003119" "NCT00003191" "NCT00003234" "NCT00003425"

[TWDGit]

Confirmed. Tested 88 "results?spons=" queries and 23 "cond=xxxx&term=xxx" queries. Everything worked perfectly. Thanks for tackling this so quickly.

[rfhb]

• Bug fix release now published at https://cran.r-project.org/package=ctrdata
• Remaining work: adapt to the new website, awaiting further details of API and analysis

[TWDGit]

Ralf. I got the error below when trying the new package today. I tried tracing the process through the ctrGetQueryUrl function but could not find an issue. I assume you are already on this, but FYI. Thanks, as always, for your work with this package. It is a HUGE help.

ScreenURL <- paste0("https://classic.clinicaltrials.gov/ct2/results?spons=%22",CoStr,"%22")
ScreenURL
[1] "https://classic.clinicaltrials.gov/ct2/results?spons=%22ACADIA+Pharmaceuticals%22"
ResultsTable <- nodbi::src_sqlite(collection="Results")

Status <- try(ctrLoadQueryIntoDb(queryterm=ScreenURL, con=ResultsTable, verbose=FALSE))
Warning: ctrGetQueryUrl(): no clinical trial register search URL found in parameter 'url' or in clipboard.
Error : 'queryterm' does not seem to result from ctrQueryHistoryInDb() or ctrGetQueryUrl(): NULL

[rfhb]

Thanks! Sorry, but I could not reproduce the error using the ScreenURL value that is above; here I found 42 records with this value using ctrLoadQueryIntoDb(ScreenURL, only.count = TRUE). Do these URLs have the search results you are interested in?

https://classic.clinicaltrials.gov/ct2/results?spons="ACADIA Pharmaceuticals"

https://classic.clinicaltrials.gov/ct2/results?spons=ACADIA+Pharmaceuticals

These work for me. For composing the string for parameter queryterm, no need to escape characters, it can be kept simple, e.g.

ScreenURL <- paste0('https://classic.clinicaltrials.gov/ct2/results?spons="', CoStr, '"')

[TWDGit]

Thanks for checking into this, Ralf. l must have messed up a dependency somewhere. I force reinstalled the package from GitHub and completely rebooted my machine. That cleared out everything. All good now. Thanks, again.

[rfhb]

Design considerations: create new ctrLoadQueryIntoDb() internal function for CTGOV API, using only JSON, based on experience with implementing this function for CTIS. Aspects for implementation:

• Resource: https://www.clinicaltrials.gov/data-about-studies/learn-about-api (current: ClinicalTrials.gov REST API 2.0.0-draft)
• Translate URLs into API request parameters, example: Search https://clinicaltrials.gov/search?cond=neuroblastoma&intr=Investigational%20drug&aggFilters=ages:child,status:com -> JSON download https://www.clinicaltrials.gov/api/v2/studies?format=json&countTotal=true&pageSize=0&query.cond=neuroblastoma&query.intr=Investigational%20Drug&aggFilters=ages%3Achild%2Cstatus%3Acom
• Pagination based on token (use pageToken as nextPageToken, pageSize is max 1000)
• Include document downloading (as for CTIS)
• No batch FHIR JSON download available (yet; https://www.clinicaltrials.gov/data-about-studies/fhir; example: https://clinicaltrials.gov/api/int/studies/download/NCT00072358?format=fhir.json)

Further questions:

• Mapping of classic and new fields, for CTGOV or across registers, at least at top levels, targeting FHIR? (Consider recently refactored dfMergeVariables(), automated schema extraction, search for relevant mapping tools, review e.g. https://json-ld.org/ with https://www.w3.org/TR/json-ld-framing/, https://github.com/dvdln/jsonpath-object-transform)
• Introduce an dbUpdate() function that iterates for identifiers and registers?
• How to hand data for a trial having been loaded from both classic and new CTGOV?
• Can ctrLoadQueryIntoDb() for EUCTR and ISRCTN also be changed to JSON?
• Create roadmap in GitHub?

[rfhb]

Check out the branch ctgov2023api for compatibility with CTGOV REST API 2.0.0-test and CTGOV Classic web interface. Install in R with

devtools::install_github(repo = "rfhb/ctrdata",  ref = "ctgov2023api", build_vignettes = TRUE)

Please report issues here.

[rfhb]

Completed with release https://github.com/rfhb/ctrdata/releases/tag/v1.15.0