Gets the same items many times

[VladimirAlexiev]

http://panic.image.ntua.gr:9876/foodanddrink/oai?verb=ListRecords&set=1003&metadataPrefix=rdf is a small set with 498 items.
http://validator.oaipmh.com/#ListRecords gets that many items.

But when I get them with Catmandu, it iterates forever. It gets the 498 items, and then repeats (not all the items in a loop, it's a more complicated pattern).

The script at the bottom gets all items (you have to abort it at some point):

alinari.pl > alinari.rdf

Then I find the item IDs like this:

perl -ne 'print qq{$1\n} if m{<edm:ProvidedCHO rdf:about="(.*?)"}' alinari.rdf > ids

and analyze them with something like

sort ids|uniq -c|sort -nr|less

In my case different items were repeated 13,12, and 1 times

---

#!perl -w

use Catmandu::Importer::OAI; # https://metacpan.org/source/HOCHSTEN/Catmandu-OAI-0.08/README
use XML::Struct::Writer; # https://metacpan.org/source/XML::Struct::Writer

my $importer = Catmandu::Importer::OAI->new
  (url => "http://panic.image.ntua.gr:9876/foodanddrink/oai",
   metadataPrefix => "rdf",
   set => 1003,
   handler => "struct", # https://metacpan.org/source/HOCHSTEN/Catmandu-OAI-0.08/lib/Catmandu/Importer/OAI/Parser/raw.pm
   # xslt => "oai-unpack.xsl",
  );

print <<EOF;
<?xml version="1.0" encoding="UTF-8"?>
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dcterms="http://purl.org/dc/terms/"
    xmlns:edm="http://www.europeana.eu/schemas/edm/"
    xmlns:oai="http://www.openarchives.org/OAI/2.0/"
    xmlns:ore="http://www.openarchives.org/ore/terms/"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#">
EOF

my $writer = XML::Struct::Writer->new (to => \*STDOUT); # , xmldecl => 0
$importer->each(sub {
  my $item = shift->{_metadata}[2]; # skip rdf::RDF and its attributes (namespaces)
  #use Data::Dump; dd $item;
  map {$writer->writeElement($_)} @$item; # multiple XML elements
  print "\n";
});

print qq{</rdf:RDF>};

[VladimirAlexiev]

Maybe that server (MINT OAI) doesn't return unique record/header/identifier. But the pagination (resumption token) protocol works fine, since http://validator.oaipmh.com/#ListRecords gets 498 records

[phochste]

Hi, yes. I can repeat the same thing here on my development box and will try to find a solution for this.