washopenresearch 0.5.0

This release makes the build of the datasets reproducible and prepares
monthly data updates. The data of the six datasets of 0.4.0 changes only
through the fixes listed below, and aqua joins as the seventh dataset.
No new journal issues are added yet; the first monthly update follows as
0.6.0.

Coverage

Five datasets are updated monthly from now on. uncnewsletter and
datapapers are frozen: still built with every release, no longer
updated.

Dataset	Updates	Articles	Years	Latest issue
washdev	monthly	1,173	2011 to 2026	Vol. 16 Issue 6
ws	monthly	4,884	2001 to 2026	Vol. 26 Issue 6
jwh	monthly	2,013	2003 to 2026	Vol. 24 Issue 6
aqua	monthly	1,819	1998 to 2026	Vol. 75 Issue 6
ploswater	monthly	434	2022 to 2026	Vol. 5 Issue 7, published up to 2026-07-06
uncnewsletter	frozen	173	2020 to 2023	newsletter ceased in May 2024
datapapers	frozen	8	2018 to 2025	harvested on 23 July 2026

Reproducible build

• The datasets are built by a targets pipeline from the raw snapshots and
  decision sheets committed in data-raw/, with the package versions
  pinned in renv.lock. One command rebuilds everything:
  Rscript data-raw/build.R. Manual corrections moved from the code into
  decision sheets keyed on the article identifier.

• A validation gate checks every dataset before anything is written:
  documented columns, a unique key, no email addresses or credentials,
  statement text wherever a statement is flagged, UN country names, and no
  article lost since the last release.

• A workflow on GitHub rebuilds the datasets on every push and pull request
  and fails when they differ from the committed data
  (data-raw/check_reproducible.R).

• Zenodo files the release as a dataset in the openwashdata community,
  from the new .zenodo.json. The releases 0.1.0, 0.3.0 and 0.4.0 were
  never archived there.

New features

• New dataset aqua: all 1,819 articles of the journal AQUA - Water
  Infrastructure, Ecosystems and Society in the journal's online archive,
  1998 to 2026, with the same columns as ws and jwh. 539 articles carry
  a data availability statement; 529 of these map to a shared das_type
  and the remaining 10 keep their full text and are listed in
  data-raw/aqua-das-review.csv.

  The 0.4.0 notes said AQUA needed a re-scrape because its statement text
  had never been captured. That was wrong. The statements were in the raw
  snapshot all along. The build read the snapshot with guessed column
  types, the journal's first statement sits beyond the rows a guess looks
  at, so the statement column was read as logical and came back empty. The
  build now states every column type (#52).

Bug fixes

• washdev: 33 correspondence author countries were three letter ISO codes
  ("IND", "GBR") in a column that otherwise holds United Nations country
  names. 32 are now the UN name. The 33rd, an affiliation in Taiwan, is
  missing, as it already was in the first author column: the UN names have
  no entry for it. The article is listed in
  data-raw/washdev-country-review.csv.

• ploswater: two articles appeared twice
  (10.1371/journal.pwat.0000024 and 10.1371/journal.pwat.0000520), each as
  two rows identical in every column. The dataset now has 434 rows, one
  per article. The cause was a search query paged without a sort order.

• Credentials are removed from the text of the datasets. Where a statement
  linked a restricted Zenodo record through an access token, the token
  parameter is now dropped and the plain record URL kept; up to v0.4.0 the
  URL kept the parameter with a placeholder value and did not resolve
  (ploswater, one article). Where authors wrote a password into their
  statement, such as the login of an FTP site, the password is replaced by
  [removed] and the rest of the sentence kept (ploswater and ws, one
  statement each).

• ws: one data availability statement kept an author email address in
  das_type. The masking of addresses inside text, introduced in v0.4.0,
  skipped that column because it is a factor. The address is now masked
  there as it already was in das.

Changes

• washdev, ws, jwh: the levels of das_type now list the shared
  statement types first ("available in online repository", "in paper",
  "on request"), followed by the statements no rule mapped. Before, the
  three types sat scattered among the statements in alphabetical order.
  Values are unchanged; code that relies on the integer codes of the factor
  needs checking.

• Credentials are removed from the raw snapshots in data-raw/, as they
  already were from the datasets. The download links of supplementary files
  in washdev.csv, ws.csv, jwh.csv and aqua.csv lose their signed
  query and keep the file path (1,752 links of 1,532 articles, all expired
  since August 2026). In ploswater.csv the access tokens in the URLs of
  one statement are dropped. A password that authors wrote into their
  statement is replaced by [removed] (ploswater.csv and ws.csv, one
  statement each). The scrapers apply the same rules to new articles. The
  datasets are unchanged. The git history was not rewritten, so commits up
  to and including v0.4.0 still hold these values.

• The raw snapshots (washdev.csv, ws.csv, jwh.csv, aqua.csv and
  ploswater.csv) no longer have the columns first_author_email and
  correspondence_author_email, and the scrapers no longer collect author
  email addresses. Contact addresses that some article pages print after
  the affiliation are masked in the same way as in the datasets
  (washdev.csv 10 articles, aqua.csv 4), and the scraper masks them in
  new articles. No other value in the snapshots changed, and the datasets
  are unchanged.

  The git history was not rewritten. Commits up to and including v0.4.0
  still hold the removed values, and the archive of v0.0.1 on Zenodo holds
  the email columns of washdev.csv. Addresses that authors wrote inside a
  data availability statement stay in the raw statement text, which is the
  published statement, and are masked in the datasets as before.

Data acquisition

• The scraper of the IWA journals is incremental. A run lists the journal
  issues again from the newest known publication year, scrapes the issues
  that are missing from the raw snapshot, and reads the two most recent
  issues again to fetch articles that were added late. An
  issue that the site lists before its articles are online is asked for
  again on the next run. The functions that decide what a run fetches are
  tested without network.

• The scraper stores an issue complete or not at all, and writes no row for
  an article whose page was not read. A page counts as read only when the
  site served the kind of page that was asked for. Its "Not Found" page
  loads completely too and was taken for an article without authors,
  statement or supplement, or for an issue without articles.

• washdev is scraped by the same scraper as ws, jwh and aqua
  (data-raw/iwa_scraping.R). The separate data-raw/washdev_scraping.R
  and the runner data-raw/run_iwa_scrapes.R are removed. The old washdev
  scraper read a throttled page as "no more issues" and stored an
  article that failed to load as an empty row. The index column that the
  first Python scraper left in data-raw/washdev.csv is removed. The
  dataset washdev is unchanged.

• The signed download links of supplementary files, which work for about
  three weeks, are written to data-raw/private/, which is not committed.
  data-raw/suppfiles_download.R reads them there.

• One command runs the monthly acquisition for every live source:
  Rscript data-raw/update_sources.R. It runs the four IWA journals in
  sequence with a cooldown, then PLOS Water, each as its own R process. A
  source that fails does not stop the others, and data-raw/update-log.csv
  records per journal and run how many journal issues and rows were added.

• The PLOS Water downloader can run incrementally. It failed on the second
  run, because it read the existing snapshot with guessed column types and
  could not combine the publication date with the new rows. It now pages
  the search in a fixed sort order and keeps one row per DOI. The two
  repeated rows are removed from data-raw/ploswater.csv; the dataset has
  been without them since the fix listed above. The unsorted paging had
  also skipped two articles (10.1371/journal.pwat.0000223 and
  10.1371/journal.pwat.0000302). They enter the dataset with the next data
  update.

• A workflow opens an issue on the 5th of each month with the checklist of
  the monthly data update (data-raw/monthly-data-update.md).

Documentation

• The help pages and the README say which datasets are updated monthly and
  which are frozen, and the README has a coverage table that is computed
  from the datasets.

• uncnewsletter: the citations column is now described in the help page
  and the data dictionary. It has been in the dataset since its first
  release without documentation. The source of the counts is not recorded.