scopusflow 0.4.0

A release about reporting a search and about trusting the cache. The package now
writes a search up as the reproducible record a systematic review has to publish,
and a family of silent cache defects that served the wrong records is fixed.

Writing up a search

• scopus_search_report() assembles the reproducible search-strategy record a
  systematic review has to report, from a harvest or from a plan not yet run. It
  prints as a readable report, formats as a methods paragraph fit to paste into a
  manuscript (format(report, style = "paragraph")), and writes as Markdown when
  file is supplied. It carries the database and platform, the field-wrapped
  query of every cell, the year limits, view, page size and paging mode, the date
  searched, the records retrieved against the number the API reported as matching,
  per-cell completeness, duplicates removed, the software versions, a runnable
  snippet that rebuilds the plan, and an explicit map of which PRISMA-S items the
  package can supply and which only you can. The reporting standard is PRISMA-S
  (Rethlefsen et al., 2021, Systematic Reviews, 10, 39), with the identification
  counts of the PRISMA 2020 flow diagram.

  Because the output is written for a published methods section, the report states
  only what the objects record. An absent retrieval time is never replaced by the
  current one, a completeness figure is never given for a harvest whose reported
  total is unknown, duplicates are counted only where a merge recorded removing
  them, and every unverifiable field says it is unrecorded, in words. Ten of the
  sixteen PRISMA-S items, among them peer review of the strategy, grey literature
  and any other database searched, are listed as yours to supply, since the
  package has no evidence for them.

• scopus_fetch_plan() now carries the per-cell accounting onto its result, as
  the cell_totals attribute, and sums it into total_results when (and only
  when) every cell reported a total. A cell that comes back shorter than the API
  said it should now warns, which the Python twin has done since 0.3.0, and the
  warning text is byte-identical between the two. A cell stopped by max_results
  is short by request and does not warn.

• scopus_combine() records the merge in the combined attribute: how many
  records went in, how many were kept, how many were removed and whether
  de-duplication ran. That count exists only at the moment of the merge, and
  PRISMA-S asks for it.

• A retrieval also records how it was paged, as the paging attribute, since an
  offset-paged query stops at the API's ceiling where a cursor-paged one does not.

Cache defects that returned the wrong records

All were silent, and all are fixed.

• A year-partitioned plan could be served another plan's years, because checkpoints
  were keyed by the cell's position alone and every cell of such a plan carries the
  same query (the year travels separately, as the API's date parameter), so the guard
  meant to reject a foreign checkpoint could not tell 2015's from 2016's. A plan over
  2016:2017 pointed at a cache written by a plan over 2015:2016 returned the 2015
  records and reported "loaded from cache". Checkpoints are now named
  cell-NNN-YYYY.rds and carry a manifest (query, date, view, page size and record
  cap) that resume compares against. Existing caches become invisible and refetch once,
  which spends a little quota to buy back correctness.
• A cell cached under a max_results cap was served to a later request asking for
  more. The manifest now records the cap and whether it actually bit, so a truncated
  checkpoint is a miss for a wider request while an untruncated one stays usable for any
  request. The Shiny app's cache key gained the record cap and now hashes the query,
  where it used to truncate it at 80 characters, so two long queries sharing a prefix no
  longer share a directory.
• A zero-row checkpoint answered for any query. The guard that recognises a foreign
  checkpoint reads the cached rows' query column, which an empty cell has no rows to
  carry, and the manifest comparison never consulted its own copy of the query, so a
  plan that legitimately found nothing left a checkpoint that a different plan pointed
  at the same cache_dir would load as its own empty result. The manifest's query must
  now match as well.
• A failed abstract retrieval was checkpointed as if it were data. A rate-limited,
  server-error or offline attempt in scopus_abstract() degrades to an all-NA row so
  the batch survives, but that row was written to the cache, so a resumed run read the
  failure back from disk and never retried it. Only rows parsed from successful
  responses are checkpointed now.
• Two identifiers could share one abstract checkpoint, because the cache filename
  reduces every non-alphanumeric character to _, so distinct identifiers such as
  10.1/a.b and 10.1/a-b collide on the same file, and resume never asked which
  identifier a cached row belonged to. The cached row's own id must now match, and a
  collision warns and refetches.
• A checkpoint holding more records than the current max_results asks for is now
  served trimmed to that cap, with the fuller set left on disk for a later, wider
  request. The opposite direction, a checkpoint truncated below what is asked for,
  already refetched with a warning.

A rejected or damaged checkpoint now warns, where it used to be mentioned only under
verbose, and checkpoints are written to a sibling temporary file and renamed into
place, so an interrupted run can no longer leave a half-written file that breaks every
later resume.

Other fixes

• scopus_combine() handed the merged set one input's own retrieval attributes.
  rbind() keeps those of its first argument, so the result inherited that harvest's
  plan, cell_totals and total_results even though it held the rows of several,
  contradicting the documented promise that per-retrieval attributes are not carried
  over. A search record read off the inherited total then called the union of two
  harvests complete against a figure belonging to one of them. Only what survives a
  merge is kept now: the earliest retrieval time, every contributing version, the
  paging mode where the inputs agree, and the merge counts themselves.
• The PRISMA 2020 identification block counted the records identified after
  de-duplication, so the two figures it gives could not both be right: the diagram
  subtracts the duplicates removed from the records identified to reach the records
  screened, and 138 identified less 11 removed is not the 138 rows the set holds. Where
  a merge recorded how many records went into it, that is now the identification count.
• A plan not yet run was described in the past tense, and its date line said the set did
  not carry a retrieval time, when the truth was that no retrieval had happened.
• average_comparison_percentage summed the numerator over years the denominator
  excluded, so a year with a missing reference count inflated it, giving a figure that
  contradicted the per-year shares printed beside it and that drives the topic ordering
  in plot_scopus_comparison(). It is now computed over the years where both counts are
  available.
• scopus_abstract(include = "references") aborted the whole batch when one document's
  response omitted its reference-count attribute. Because that arrived as a bare base
  error, untyped and so outside the scopus_error hierarchy, the per-identifier handler
  could not catch it, defeating the documented promise that one bad identifier does not
  lose a batch.
• The app's demo banner still called the demo data synthetic, which was true of the
  six invented rows that shipped before 0.3.0. Demo mode has replayed 138 real
  published articles since then, and only the topic comparison is simulated, so the
  banner now says which is which.
• A record count past 2^31 was coerced to NA at three points, each of which
  accepted any finite whole number and then reached for a 32-bit integer. scopus_top(n = )
  left head.data.frame() raising an untyped error about its own argument;
  scopus_fetch(max_results = ) and scopus_fetch_plan(max_results = ) turned every
  later n >= max_results into NA, so the retrieval died on a bare "missing value
  where TRUE/FALSE needed"; and the app's reproducible-code panel wrote
  max_results = NA into a script it promises is runnable. All three now carry the
  value as a double, so a cap wider than the result set simply never bites, which is
  what the Python twin has always done. A cap that fits stays an integer.
• The 5000-record cap warning fired even when max_results had already asked for fewer
  records than the ceiling, advising a remedy for a problem the caller did not have.
• scopus_compare_topics() carries its counts as doubles, matching scopus_trend() and
  scopus_intersections(), so billion-scale totals no longer coerce to NA.
• Every request carries a 60-second transfer timeout, tunable through
  getOption("scopusflow.timeout"), so a stalled connection cannot hang a session.
• Retrievals record when they were taken and by which package version, as the
  retrieved_at and scopusflow_version attributes, preserved through the .rds round
  trip. Both are attributes, so the documented schema and the CSV round trip are
  unchanged.
• Every text file the package writes now carries LF line endings on all platforms: the
  BibTeX and RIS exports, the DOI and record CSVs, and the app's script and comparison
  downloads previously arrived with CRLF endings when written on Windows.
• Every print() method now sends its whole output to standard output. The header line
  went through cli, which writes to the message stream, so capture.output() returned
  th...

scopusflow 0.3.0

The bundled example records become a real literature. example_records now holds 138 real journal articles on graphene supercapacitors from 2015 to 2024, replacing the six invented rows, and every vignette and example runs on it beside the key-gated live call a reader would actually write.

The records cannot come from Scopus, whose API terms do not permit redistributing retrieved records, so they come from OpenAlex under CC0, reshaped into the schema scopus_fetch() returns. The help page and the design notes state that provenance. The harvest is complete rather than sampled, so the rows per year are the real publications per year; eleven records carry no DOI and two no source title, kept because a real harvest has such gaps.

The parser fixture moves onto the reserved 10.5555 example prefix, having previously paired genuine, resolving DOIs with invented titles.

R CMD check --as-cran: 0 errors, 0 warnings. Not yet submitted to CRAN, where 0.1.0 remains current.

scopusflow 0.2.1

A documentation release. The vignettes now demonstrate several features that 0.2.0 shipped but did not show: the AND NOT operator, a complete field-tagged intersections concept, scopus_top(), verbose fetch-plan progress, extract_dois() deduplication forms and a corpus keyword tally.

See NEWS.md for the full list.

scopusflow 0.2.0

The second release: author keywords and structured reference retrieval, corpus assembly for downstream tools, concept-intersection sizing and plotting, cursor pagination past the 5000-record ceiling, reference-manager export, the code-free app, and hardened caching. Full details in NEWS.md.

scopusflow 0.1.0

Initial release, available on CRAN: https://cran.r-project.org/package=scopusflow

First release.

• Reproducible search plans with scopus_plan(), and cheap sizing with
  scopus_count().
• Quota-aware, paginated retrieval through scopus_fetch(), with the largest
  page each view allows requested by default to keep request counts low, and
  resumable, cached, partitioned retrieval through scopus_fetch_plan().
• A stable normalised record schema from scopus_records(), with a
  summary() method that gives a quick overview.
• DOI extraction and change tracking with scopus_extract_dois() and
  scopus_diff_dois().
• Topic-trend comparison with scopus_compare_topics(), and a plot from
  plot_scopus_comparison() or autoplot().
• Interoperability and I/O through as_bibliometrix(),
  write_scopus_records() and read_scopus_records().
• A reference to the common 'Scopus' field tags in scopus_field_tags(), a
  safe query composer in scopus_query(), and a bundled example_records
  dataset for offline exploration.
• Safe merging of record sets with scopus_combine() (and a c() method), plus
  as_tibble() and as.data.frame() coercion.
• A typed condition system (scopus_error and its subclasses) and quota-header
  parsing with scopus_quota().
• The comparison plot uses whole-number year breaks, a colour-blind-safe palette,
  direct line labels, an optional highlight argument and a shaded Wilson
  stability band (an illustrative range, switchable with interval).
• The bundled example_records spans several disciplines, and the examples and
  five workflow vignettes draw on a wide range of fields.
• Multiple authors are retained in the authors column rather than truncated to
  the first; very large result totals are handled without overflow; and DOI
  cleaning copes with www.doi.org hosts and DOI: labels.