Producer: on adding 200 repos, only 178 are in DB

[bzz]

• borges producer --source=file --file ./top200repos.txt
• got 200 messages in Rabbit, borges.buriedQueue empty

   curl -s -u guest:guest "http://localhost:8081/api/queues/%2F/borges" | jq .messages
  

• got 178 records in DB

   select count(1) from repositories;
  

top200repos.txt
200 lines files, cat top200repos.txt | sort -u | uniq -d -c is empty so there is no duplicated and wc -l is 200

[bzz]

Same happens with 2000 repos from text file:

• queue has 2000 elements,
• but DB only 1814

testing=# select count(*) from repositories;
 count
-------
  1814

[ajnavarro]

@bzz is not a Borges error, we have duplicated repos in python and java lists, like this: github.com/mihaic/graphalytics.git

[ajnavarro]

@bzz Also you can check it in the top200repos.txt file too: sort top200repos.txt | uniq --count.

I will close the issue, feel free to reopen if I'm wrong.

[bzz]

Thanks for catching this! I believe the confusion is from sort -u above, wich already filters out all dupes.

✅ sort top200repos.txt | uniq | wc -l 178
✅ sort top200repos.txt | uniq | wc -l 1814