add items

[sjgknight]

Hi
Apologies if I'm missing something, I'm trying to add items to Zotero via a tibble of items. I've checked the item structure in c2z, and mutated my tibble (from openalex, using openalexr package) to match minus the Zotero metadata fields (e.g., ID).
There's no obvious error in the c2z::ZoteroPost log.
I can provide a reprex issue, but realise this is unlikely to be priority (and thanks for creating and sharing the package).

group_library <- Zotero(user = F, id = group_id, library = TRUE, silent = TRUE, index = F)
target_collection <- group_lib$collections$key[group_lib$collections$name == "reading"]

new_collect <- Zotero(
  zotero = group_lib,
  collection.names = c(paste0("priority_", Sys.Date())),
  library = T,
  create = T 
) 

new_collect$collections <- new_collect$collections %>%
  dplyr::mutate(parentCollection = dplyr::case_when(name == paste0("priority_", Sys.Date()) ~ target_collection,
                                                    .default = parentCollection))
  
c2z::ZoteroPost(
  new_collect,
  post.collections = T
)


# Openalex inherits MAG I think, so in theory https://github.com/zotero/translators/blob/master/Microsoft%20Academic.js should give some guidance 
items <- literature_priority %>%
    unnest_wider(col = c(author, concepts), names_sep = "_") %>%
      rowwise() %>%
    mutate(
        creators = list( #needs to be a tibble, creatorType, firstName, lastName 
            data.frame(
                creatorType = "author",
                firstName = sapply(author_au_display_name, function(names) sub(" .*", "", names)),
                lastName = sapply(author_au_display_name, function(names) sub(".* ", "", names))
            )
        ),
        tags = list( #needs to be tag, type (set type to 1 for them all)
            data.frame(
                tag = concepts_display_name,
                type = 1
            )
        )) %>%
  transmute(title = display_name,
         abstractNote = ab, 
         date = publication_date,
         rights = license, 
         url = ifelse(!is.na(oa_url), oa_url, url),
         language = language,
         DOI = doi,
         extra = paste0("grants:", grants, "\n",
               "cited_by_count:",  cited_by_count, "\n",
               "counts_by_year:", counts_by_year, "\n",
               "referenced_works:", referenced_works, "\n",
               "related_works:", related_works, "\n",
               "is_paratext:", is_paratext, "\n",
               "is_retracted:", is_retracted, "\n", collapse = "\n"),
         pages = paste0(first_page, "-", last_page),
         numPages = as.numeric(last_page) - as.numeric(first_page),
         volume = volume,
         issue = issue,
         ISSN = issn_l,
         creators = list(creators),
        itemType = case_when(type == "article" ~ "journalArticle",
                             type == "dissertation" ~ "thesis",
                             type == "book-chapter" ~ "bookSection",
                             .default = "document"),
         tags = list(tags),
         publisher = host_organization, 
         university = ifelse(type == "dissertation", host_organization, ""),
         bookTitle = ifelse(type == "book-chapter", so, ""),
         publicationTitle = ifelse(type == "article", so, ""),
        collections = c(new_collect$collection.key)
         )

post_items <-  Zotero(
  zotero = new_collect,
  items = items,
  library = T,
  create = T 
) 


zlog <- c2z::ZoteroPost(
  post_items,
  post.items = T,
  post.collections = T
)

[oeysan]

Hi, Simon!

Thanks for making me aware of this great resource, I'd not heard of openalex prior to your issue!

Concerning your issue, I apologize that the documentation is lacking (and the code is a mess). I've pushed a revised version of the package, and it should hopefully cover your problem.

# Check that the items from openalexR as Zotero-friendly
items <- ZoteroFormat(items, check.structure = TRUE)

post_items <- Zotero(
  zotero = new_collect,
  library = TRUE, 
  metadata = items,
  post = TRUE
) ```

[sjgknight]

Great! It's pretty good, and should cover crossref+ (I've been playing with it as an alternative to the closed indexes), there have been a couple of studies on its coverage. There are some disambiguation issues with it (I know some papers under my name were not authored by me), but that was also true of MAG, and I hope they'll be able to fix it.

Thanks for the quick reply too, and for sharing the package. I tested, and I can see using ZoteroFormat adds key and version values, but:

1. it seems to also be removing my creators column, whether it's contains named lists or nested tibbles.
2. possibly related, I don't seem to be adding the items even in the local data using the Zotero function (so, ZoteroPush then also fails).

This is a single row of openalexR form data (lightly edited) pre-wrangling:

literature_priority %>% head(n=1) %>% datapasta::df_paste()

data.frame(
             stringsAsFactors = FALSE,
                           id = c("https://openalex.org/W2038117241"),
                                                   display_name = c("The ‘everywhere and nowhere’ nature of thinking as a subject-specific competency",
                                  NA),
                                                         author = c(list(au_id = "https://openalex.org/A5076633594", au_display_name = "Rosemary Hipkins", au_orcid = "https://orcid.org/0000-0003-3750-2488", author_position = "first", au_affiliation_raw = "New Zealand Council for Educational Research , New Zealand", institution_id = "https://openalex.org/I2799883461", institution_display_name = "New Zealand Council for Educational Research", institution_ror = "https://ror.org/05jdvx041", institution_country_code = "NZ", institution_type = "government", institution_lineage = "https://openalex.org/I2799883461"),
                                  NA),
                                                             ab = c("Abstract some text here.",
                                  NA),
             publication_date = c("2013-12-01"),
                           so = c("Thinking Skills and Creativity"),
                        so_id = c("https://openalex.org/S42886211"),
            host_organization = c("Elsevier BV"),
                       issn_l = c("1871-1871"),
                          url = c("https://doi.org/10.1016/j.tsc.2013.05.006"),
                      pdf_url = c(NA),
                      license = c(NA),
                   first_page = c("221"),
                    last_page = c("232"),
                       volume = c("10"),
                        issue = c(NA),
                       oa_url = c(NA),
                     language = c("en"),
                       grants = c(NA),
               cited_by_count = c(4L),
                                                 counts_by_year = c(list(year = c(2022, 2020, 2017, 2016), cited_by_count = c(1, 1, 1, 1)),
                                  NA),
             publication_year = c(2013L),
                                               cited_by_api_url = c("https://api.openalex.org/works?filter=cites:W2038117241",
                                  NA),
                                                            ids = c(c(openalex = "https://openalex.org/W2038117241", doi = "https://doi.org/10.1016/j.tsc.2013.05.006", mag = "2038117241"),
                                  NA),
                          doi = c("https://doi.org/10.1016/j.tsc.2013.05.006"),
                         type = c("article"),
                                               referenced_works = c(c("https://openalex.org/W31794006", "https://openalex.org/W341287568", "https://openalex.org/W1976178032", "https://openalex.org/W1982167621", "https://openalex.org/W2009368397", "https://openalex.org/W2047762721", "https://openalex.org/W2048306610", "https://openalex.org/W2065557597", "https://openalex.org/W2078861360", "https://openalex.org/W2089608703", "https://openalex.org/W2119116459", "https://openalex.org/W2128340945", "https://openalex.org/W2167705687", "https://openalex.org/W2256997535", 
"https://openalex.org/W3045877044", "https://openalex.org/W3080302530", "https://openalex.org/W4249395625"),
                                  NA),
                                                  related_works = c(c("https://openalex.org/W2085642841", "https://openalex.org/W2093287941", "https://openalex.org/W2349943118", "https://openalex.org/W2363900944", "https://openalex.org/W2367458741", "https://openalex.org/W2373580576", "https://openalex.org/W2748952813", "https://openalex.org/W3004287565", "https://openalex.org/W3121682328", "https://openalex.org/W3200122776"),
                                  NA),
                  is_paratext = c(FALSE),
                 is_retracted = c(FALSE),
                                                       concepts = c(list(id = c("https://openalex.org/C2777855551", "https://openalex.org/C15744967", "https://openalex.org/C180747234", "https://openalex.org/C145420912", "https://openalex.org/C41008148", "https://openalex.org/C161191863"), wikidata = c("https://www.wikidata.org/wiki/Q12310021", "https://www.wikidata.org/wiki/Q9418", "https://www.wikidata.org/wiki/Q23373", "https://www.wikidata.org/wiki/Q853077", "https://www.wikidata.org/wiki/Q21198", "https://www.wikidata.org/wiki/Q199655"), display_name = c("Subject (documents)", 
"Psychology", "Cognitive psychology", "Mathematics education", "Computer science", "Library science"), level = c(2, 0, 1, 1, 0, 1), score = c(0.7253356, 0.5831753, 0.36041608, 0.3462116, 0.18166196, 0)),
                                  NA)
)

Post-wrangling:

data.frame(
  stringsAsFactors = FALSE,
                                    title = c("The ‘everywhere and nowhere’ nature of thinking as a subject-specific competency",
                       NA),
                             abstractNote = c("Abstract text here.",
                       NA),
              date = c("2013-12-01"),
            rights = c(NA),
               url = c("https://doi.org/10.1016/j.tsc.2013.05.006"),
          language = c("en"),
               DOI = c("https://doi.org/10.1016/j.tsc.2013.05.006"),
                                    extra = c("not priority and a bit of a mess"),
             pages = c("221-232"),
          numPages = c(11),
            volume = c("10"),
             issue = c(NA),
              ISSN = c("1871-1871"),
                                 creators = c(list(creatorType = "author", name = "Rosemary Hipkins", firstName = "Rosemary", lastName = "Hipkins"),
                       NA),
          itemType = c("journalArticle"),
                                     tags = c(list(tag = c("Subject (documents)", "Psychology", "Cognitive psychology", "Mathematics education", "Computer science", "Library science"), type = c(1, 1, 1, 1, 1, 1)),
                       NA),
         publisher = c("Elsevier BV"),
        university = c(NA),
         bookTitle = c(NA),
  publicationTitle = c("Thinking Skills and Creativity"),
  parentCollection = c("CKTXHD5K"),
       collections = c("CKTXHD5K")
)

Tested both:

item_form <- c2z::ZoteroFormat(items, check.structure = TRUE)

items <- items %>% select(title, creators) %>% dplyr::left_join(item_form, by = "title") 
items <- items %>% 
  mutate(creators = list(flatten(lst(creators)))) 

items <- c2z::ZoteroFormat(items, check.structure = TRUE)

Then as I understand, to add these to Zotero I first add them to the local data structure using this, where group_lib is just selected with an ID.

group_lib <- Zotero(user = F, id = group_id, library = TRUE, silent = TRUE, index = F)

post_items <-  Zotero(
  zotero = group_lib,
  items = items,
  library = T,
  create = T
) 

And then to post to the API, as with the collection (and here assigned so I can inspect the log), this bit won't work at the moment because post_items isn't functioning.

zlog <- c2z::ZoteroPost(
  post_items,
  post.items = T,
  post.collections = T
)

[oeysan]

Can you check the lastest commit? It hope it should solve the problem, but it needs further testing.
I was planning to remove the log system in the functions, as it was more an afterthought than anything else, and replace it with codecov, but I've not found the time for it yet.

[sjgknight]

Using the example above, ZoteroFormat(items, check.structure = T) now looks to return well formed data.

Issue: ZoteroAdd(group_lib, metadata = items) seemed to replace the collection value with the full list of collection keys for the appended items.
fix: group_lib$items <- c2z:::AddAppend(items, group_lib$items) appends the items correctly.

I made one minor change to my earlier mutate, to remove the rownames from creators (zotero rejects it otherwise):

creators = list( #needs to be a tibble, creatorType, firstName, lastName 
            data.frame(
                creatorType = "author",
                name = author_au_display_name,
                firstName = sapply(author_au_display_name, function(names) sub(" .*", "", names)),
                lastName = sapply(author_au_display_name, function(names) sub(".* ", "", names))
            ) %>% 
              tibble::remove_rownames()
        )

Then this works to post them:

zlog <- c2z::ZoteroPost(
  zotero = group_lib,
  post.items = T,
  post.collections = T
)

Thanks!