Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thorium Reader is not able to borrow books from an OPDS feed #1148

Closed
vbessonov opened this issue Aug 18, 2020 · 20 comments · Fixed by #1275
Closed

Thorium Reader is not able to borrow books from an OPDS feed #1148

vbessonov opened this issue Aug 18, 2020 · 20 comments · Fixed by #1275
Assignees
Projects

Comments

@vbessonov
Copy link

vbessonov commented Aug 18, 2020

I faced with an issue when Thorium Reader is not able to borrow books from an OPDS feed.

The OPDS feed I'm using: https://github.com/vbessonov/opds-feed

Steps to reproduce:

  1. git clone https://github.com/vbessonov/opds-feed
  2. cd opds-feed
  3. docker-compose up -d
  4. Open Thorium Reader and add a new OPDS feed using http://localhost:8080 as its URL:
    image
  5. Open Test OPDS feed catalog and try to borrow a book:
    image
  6. Thorium Reader opens a borrow link in a browser and then stops:
    image

The books wasn't downloaded and isn't accessible in Thorium Reader:
image

@danielweck
Copy link
Member

Thank you very much for the detailed description of the problem.

Thorium does not support PDF publications, so this could be the cause of the behaviour. Would you be able to test with an OPDS feed that serves EPUB publications?

@vbessonov
Copy link
Author

Absolutely, will do

@CircularKen
Copy link

Should that be "Thorium does not yet support PDF publications"…?

@vbessonov
Copy link
Author

vbessonov commented Aug 18, 2020

@danielweck I tried with an EPUB and updated my repository accordingly but it worked the same way, I wasn't able to download the book

@panaC
Copy link
Member

panaC commented Aug 18, 2020

@vbessonov publish the log, please

@vbessonov
Copy link
Author

@panaC, sure but I downloaded the Thorium's binary from the website, could you tell me how I can get the logs?

@vbessonov
Copy link
Author

Unfortunately, it didn't work. Basically, http://localhost:8080/fulfill/ gets rewritten by nginx to http://localhost:8080/fulfill/index.epub anyways

@vbessonov
Copy link
Author

@panaC, is there anything else I can do to make it work?

@vbessonov
Copy link
Author

@danielweck, what can I do to speed up the investigation process?

@danielweck
Copy link
Member

Sorry, I have been out of office.
You could try running Thorium from its source code (develop branch):

  1. git clone https://github.com/edrlab/thorium-reader.git
  2. cd thorium-reader
  3. npm install (this will take a while ... so grab your drink of choice)
  4. npm run start:dev (or SKIP_CHECK_TYPESCRIPT=1 npm run start:dev to speed up the compilation process)

@danielweck
Copy link
Member

So far I am not seeing anything suspicious in the OPDS server:

curl -s -L -I -X GET "http://localhost:8080/borrow"
=>

HTTP/1.1 301 Moved Permanently
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:23:16 GMT
Content-Type: text/html
Content-Length: 169
Location: http://localhost/borrow/
Connection: keep-alive

curl -s -L -I -X GET "http://localhost:8080/borrow/index.xml"
=>

HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:23:30 GMT
Content-Type: application/atom+xml
Content-Length: 2931
Last-Modified: Fri, 21 Aug 2020 17:17:27 GMT
Connection: keep-alive
ETag: "5f4001a7-b73"
Accept-Ranges: bytes

@danielweck
Copy link
Member

danielweck commented Aug 21, 2020

EDIT --- see message below (there is a typo in the OPDS feed)


@panaC I am seeing this in the console:

(note that I added http://localhost:8080/borrow/index.xml directly to the Catalogs, instead of http://localhost:8080/, and I also tried changing the href from http://localhost:8080/fulfill/ to http://localhost:8080/fulfill/index.epub in http://localhost:8080/borrow/index.xml)

[2]   readium-desktop:main#services/publication [START] Download publication http://localhost:8080/fulfill/index.epub +3m
[2]   readium-desktop:main#services/downloader DOWNLOAD process:  http://localhost:8080/fulfill/index.epub +3m
[2]   readium-desktop:main#services/downloader oauth get +1ms
[2]   readium-desktop:main#services/downloader NotFoundError { message: 'document not found' } +0ms
[2]   readium-desktop:main#services/downloader Error while downloading resource {
[2]   identifier: '1ce5431d-1ffd-4fec-b347-6f3b54331ba5',
[2]   srcUrl: 'http://localhost:8080/fulfill/index.epub',
[2]   dstPath: '/var/folders/gm/ws1_n9s97y1b3z4vfgw79h240000gn/T/readium-desktop--23667-5N4QZxgbEXv0-.epub',
[2]   extension: '.epub',
[2]   progress: 0,
[2]   downloadedSize: 0,
[2]   status: 3
[2] } 404 +6ms
[2]   readium-desktop:sync ### action type TOAST_OPEN_REQUEST +9ms
[2]   readium-desktop:sync send to 816323f2-4cc6-4497-b4f3-30267c081898 +0ms
[2]   readium-desktop:sync ### action type DOWNLOAD_ERROR +0ms
[2]   readium-desktop:sync send to 816323f2-4cc6-4497-b4f3-30267c081898 +0ms
[2]   readium-desktop:main:saga:api API-ERROR Error: importPublicationFromOpdsDoc error Error while downloading resource: 404
[2]     at PublicationApi.<anonymous> (/Users/danielweck/Downloads/500GB/Code/readium-desktop/dist/main.js:3162:27)
[2]     at Generator.throw (<anonymous>)
[2]     at rejected (/Users/danielweck/Downloads/500GB/Code/readium-desktop/node_modules/tslib/tslib.js:112:69)
[2]     at processTicksAndRejections (internal/process/task_queues.js:94:5) requestAction:  {
[2]   type: 'API_REQUEST',
[2]   payload: [
[2]     {
[2]       url: 'http://localhost:8080/fulfill/index.epub',
[2]       title: undefined,
[2]       type: 'application/epub+zip',
[2]       properties: undefined
[2]     },
[2]     'eyJtZXRhZGF0YSI6eyJ0aXRsZSI6IkRvc3RvZXZza3kgRXB1YiIsImlkZW50aWZpZXIiOiJ1cm46aXNibjo5NzgwMjMxNTQzOTczIiwiYXV0aG9yIjoiU3BlYyBXcml0ZXIiLCJsYW5ndWFnZSI6ImVuIiwibW9kaWZpZWQiOiIyMDIwLTA4LTE3VDE1OjQ4OjQxLjAwMFoiLCJwdWJsaXNoZWQiOiIyMDIwLTA4LTE3VDAwOjAwOjAwLjAwMFoiLCJzdWJqZWN0IjpbeyJuYW1lIjoiQWR1bHQiLCJzY2hlbWUiOiJodHRwOi8vc2NoZW1hLm9yZy9hdWRpZW5jZSIsImNvZGUiOiJBZHVsdCJ9LHsibmFtZSI6Ik5vbmZpY3Rpb24iLCJzY2hlbWUiOiJodHRwOi8vbGlicmFyeXNpbXBsaWZpZWQub3JnL3Rlcm1zL2ZpY3Rpb24vIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZmljdGlvbi9Ob25maWN0aW9uIn0seyJuYW1lIjoiVGVjaG5vbG9neSIsInNjaGVtZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvVGVjaG5vbG9neSJ9LHsibmFtZSI6IkFyY2hpdGVjdHVyZSIsInNjaGVtZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvIiwiY29kZSI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvZ2VucmVzL1NpbXBsaWZpZWQvQXJjaGl0ZWN0dXJlIn1dfSwibGlua3MiOlt7InJlbCI6Imlzc3VlcyIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi93b3Jrcy9JU0JOLzk3ODAyMzE1NDM5NzMvcmVwb3J0In0seyJ0eXBlIjoiYXBwbGljYXRpb24vYXRvbSt4bWw7dHlwZT1lbnRyeTtwcm9maWxlPW9wZHMtY2F0YWxvZyIsInJlbCI6ImFsdGVybmF0ZSIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi93b3Jrcy9JU0JOLzk3ODAyMzE1NDM5NzMifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9lcHViK3ppcCIsInJlbCI6Imh0dHA6Ly9vcGRzLXNwZWMub3JnL2FjcXVpc2l0aW9uIiwiaHJlZiI6Imh0dHA6Ly9sb2NhbGhvc3Q6ODA4MC9mdWxmaWxsL2luZGV4LmVwdWIifSx7InJlbCI6Imh0dHA6Ly9saWJyYXJ5c2ltcGxpZmllZC5vcmcvdGVybXMvcmVsL3Jldm9rZSIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi9sb2Fucy8xMC9yZXZva2UifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9hdG9tK3htbDtwcm9maWxlPW9wZHMtY2F0YWxvZztraW5kPWFjcXVpc2l0aW9uIiwidGl0bGUiOiJSZWNvbW1lbmRlZCBXb3JrcyIsInJlbCI6InJlbGF0ZWQiLCJocmVmIjoiaHR0cDovL2NtLmhpbGJlcnR0ZWFtLm5ldDo2NTAwL01BSU4vd29ya3MvSVNCTi85NzgwMjMxNTQzOTczL3JlbGF0ZWRfYm9va3MifSx7InR5cGUiOiJhcHBsaWNhdGlvbi9sZCtqc29uOyBwcm9maWxlPVwiaHR0cDovL3d3dy53My5vcmcvbnMvYW5uby5qc29ubGRcIiIsInJlbCI6Imh0dHA6Ly93d3cudzMub3JnL25zL29hI2Fubm90YXRpb25TZXJ2aWNlIiwiaHJlZiI6Imh0dHA6Ly9jbS5oaWxiZXJ0dGVhbS5uZXQ6NjUwMC9NQUlOL2Fubm90YXRpb25zL0lTQk4vOTc4MDIzMTU0Mzk3MyJ9LHsicmVsIjoiaHR0cDovL2xpYnJhcnlzaW1wbGlmaWVkLm9yZy90ZXJtcy9yZWwvYW5hbHl0aWNzL29wZW4tYm9vayIsImhyZWYiOiJodHRwOi8vY20uaGlsYmVydHRlYW0ubmV0OjY1MDAvTUFJTi9hbmFseXRpY3MvSVNCTi85NzgwMjMxNTQzOTczL29wZW5fYm9vayJ9XX0='
[2]   ],
[2]   meta: {
[2]     api: {
[2]       requestId: 'SAME_FILE_IMPORT_REQUEST',
[2]       moduleId: 'publication',
[2]       methodId: 'importOpdsPublicationLink'
[2]     }
[2]   },
[2]   sender: { type: 1, identifier: '816323f2-4cc6-4497-b4f3-30267c081898' }
[2] } +10ms

@danielweck
Copy link
Member

@vbessonov there is a typo in:

https://github.com/vbessonov/opds-feed/blob/15514f87702dff9d3195c03ed4fce8af4a3d315e/content/borrow/index.xml#L29

fulfil vs. fulfill

curl -s -L -I -X GET "http://localhost:8080/fulfil/index.epub"
=>

HTTP/1.1 200 OK
Server: nginx/1.19.2
Date: Fri, 21 Aug 2020 17:51:51 GMT
Content-Type: application/epub+zip
Content-Length: 798536
Last-Modified: Fri, 21 Aug 2020 17:17:27 GMT
Connection: keep-alive
ETag: "5f4001a7-c2f48"
Accept-Ranges: bytes

@danielweck
Copy link
Member

@panaC when clicking on the EPUB cover, the popup dialog appears and shows the "borrow" link, and a couple of seconds later this gets replaced with an "import" link (presumably, there is an async HTTP fetch to a "publication" dataset, which replaces the default one found in the original OPDS feed). The "import" link works fine. During the 2 seconds gap, if I click on the "borrow" link, it can be opened in the web browser, but not in Thorium. Any idea what is going on?

@panaC
Copy link
Member

panaC commented Oct 6, 2020

  <entry schema:additionalType="http://schema.org/EBook">
    <title>Dostoevsky Epub</title>
    <author>
      <name>Spec Writer</name>
      <link href="http://cm.hilbertteam.net:6500/MAIN/works/contributor/Spec%20Writer/eng/Adult%2CAdults%2BOnly%2CAll%2BAges%2CChildren%2CYoung%2BAdult" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="contributor" title="Spec Writer"/>
    </author>
    <simplified:pwid>1ef566f9-bda4-109d-e475-729437d2b6da</simplified:pwid>
    <category term="Adult" scheme="http://schema.org/audience" label="Adult"/>
    <category term="http://librarysimplified.org/terms/fiction/Nonfiction" scheme="http://librarysimplified.org/terms/fiction/" label="Nonfiction"/>
    <category term="http://librarysimplified.org/terms/genres/Simplified/Technology" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Technology"/>
    <category term="http://librarysimplified.org/terms/genres/Simplified/Architecture" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Architecture"/>
    <dcterms:language>en</dcterms:language>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/report" rel="issues"/>
    <id>urn:isbn:9780231543973</id>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973" type="application/atom+xml;type=entry;profile=opds-catalog" rel="alternate"/>
    <bibframe:distribution bibframe:ProviderName="oapen"/>
    <published>2020-08-17T00:00:00Z</published>
    <link href="http://cm.hilbertteam.net:6500/MAIN/groups/142?entrypoint=Book" rel="collection" title="Nonfiction"/>
    <updated>2020-08-17T15:48:41Z</updated>
    <link href="http://localhost:8080/borrow/" rel="http://opds-spec.org/acquisition/borrow" type="application/atom+xml;type=entry;profile=opds-catalog">
      <opds:indirectAcquisition type="application/epub+zip"/>
      <opds:availability status="available"/>
    </link>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/related_books" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="related" title="Recommended Works"/>
    <link href="http://cm.hilbertteam.net:6500/MAIN/annotations/ISBN/9780231543973" type="application/ld+json; profile=&quot;http://www.w3.org/ns/anno.jsonld&quot;" rel="http://www.w3.org/ns/oa#annotationService"/>
    <link href="http://cm.hilbertteam.net:6500/MAIN/analytics/ISBN/9780231543973/open_book" rel="http://librarysimplified.org/terms/rel/analytics/open-book"/>
  </entry>

2 entry links are extracted from this entry :

[2]   {
[2]     url: 'http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973',
[2]     title: undefined,
[2]     type: 'application/atom+xml;type=entry;profile=opds-catalog',
[2]     properties: undefined
[2]   },
[2]   {
[2]     url: 'http://localhost:8080/borrow/',
[2]     title: undefined,
[2]     type: 'application/atom+xml;type=entry;profile=opds-catalog',
[2]     properties: {
[2]       indirectAcquisitionType: 'application/epub+zip',
[2]       lcpHashedPassphrase: undefined,
[2]       numberOfItems: undefined,
[2]       priceValue: undefined,
[2]       priceCurrency: undefined,
[2]       holdTotal: undefined,
[2]       holdPosition: undefined,
[2]       copyTotal: undefined,
[2]       copyAvailable: undefined,
[2]       availabilityState: undefined,
[2]       availabilitySince: undefined,
[2]       availabilityUntil: undefined
[2]     }
[2]   }

the first timeout and the second returns :

<entry xmlns:app="http://www.w3.org/2007/app"
  xmlns:bib="http://bib.schema.org/"
  xmlns:bibframe="http://bibframe.org/vocab/"
  xmlns:dcterms="http://purl.org/dc/terms/"
  xmlns:drm="http://librarysimplified.org/terms/drm"
  xmlns:opds="http://opds-spec.org/2010/catalog"
  xmlns:opensearch="http://a9.com/-/spec/opensearch/1.1/"
  xmlns:opf="http://www.idpf.org/2007/opf"
  xmlns:schema="http://schema.org/"
  xmlns:simplified="http://librarysimplified.org/terms/"
  xmlns="http://www.w3.org/2005/Atom" schema:additionalType="http://schema.org/EBook">
  <title>Dostoevsky Epub</title>
  <author>
    <name>Spec Writer</name>
    <link href="http://cm.hilbertteam.net:6500/MAIN/works/contributor/Spec%20Writer/eng/Adult%2CAdults%2BOnly%2CAll%2BAges%2CChildren%2CYoung%2BAdult" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="contributor" title="Spec Writer"/>
  </author>
  <simplified:pwid>1ef566f9-bda4-109d-e475-729437d2b6da</simplified:pwid>
  <category term="Adult" scheme="http://schema.org/audience" label="Adult"/>
  <category term="http://librarysimplified.org/terms/fiction/Nonfiction" scheme="http://librarysimplified.org/terms/fiction/" label="Nonfiction"/>
  <category term="http://librarysimplified.org/terms/genres/Simplified/Technology" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Technology"/>
  <category term="http://librarysimplified.org/terms/genres/Simplified/Architecture" scheme="http://librarysimplified.org/terms/genres/Simplified/" label="Architecture"/>
  <dcterms:language>en</dcterms:language>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/report" rel="issues"/>
  <id>urn:isbn:9780231543973</id>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973" type="application/atom+xml;type=entry;profile=opds-catalog" rel="alternate"/>
  <bibframe:distribution bibframe:ProviderName="oapen"/>
  <published>2020-08-17T00:00:00Z</published>
  <updated>2020-08-17T15:48:41Z</updated>
  <link href="http://localhost:8080/fulfill/" rel="http://opds-spec.org/acquisition" type="application/epub+zip">
    <opds:availability since="2020-08-17T15:52:17Z" status="available" until="2020-09-07T15:52:17Z"/>
  </link>
  <link href="http://cm.hilbertteam.net:6500/MAIN/loans/10/revoke" rel="http://librarysimplified.org/terms/rel/revoke"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/works/ISBN/9780231543973/related_books" type="application/atom+xml;profile=opds-catalog;kind=acquisition" rel="related" title="Recommended Works"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/annotations/ISBN/9780231543973" type="application/ld+json; profile=&quot;http://www.w3.org/ns/anno.jsonld&quot;" rel="http://www.w3.org/ns/oa#annotationService"/>
  <link href="http://cm.hilbertteam.net:6500/MAIN/analytics/ISBN/9780231543973/open_book" rel="http://librarysimplified.org/terms/rel/analytics/open-book"/>
</entry>

the publication information view is updated and display this new entry. There is no more a borrow link but an acquisition link. This is why the view changed.
as far as borrow link is concerned, it's not an acceptable mimeType to download it and therefore opens the web browser.

the acceptable mimeType are :

const supportedFileTypeLinkArray = [
    ContentType.AudioBookPacked,
    ContentType.AudioBookPackedLcp,
    ContentType.Epub,
    ContentType.Lcp,
    ContentType.AudioBook,
    ContentType.Json,
    ContentType.JsonLd,
];

json and jsonld are only accepted when the format is readium web publication manifest (RWPM) to package it.

I think this behaviour is standard, and i can close this issue.
feel free to reopen it.

@panaC panaC closed this as completed Oct 6, 2020
@llemeurfr llemeurfr added this to To do in v1.6 Oct 14, 2020
@llemeurfr
Copy link
Contributor

@vbessonov is it finally something NYPL has to correct in its feeds?

@llemeurfr llemeurfr removed this from To do in v1.6 Oct 14, 2020
@panaC panaC reopened this Dec 3, 2020
@panaC
Copy link
Member

panaC commented Dec 3, 2020

My bad,
It's an issue with opds1 . I think.
Borrow link isn't an alternate opdsEntry.

const entrylinkView = fallback(
this.convertFilterLinkToView(baseUrl, r2OpdsPublication.Links, {
type: "type=entry;profile=opds-catalog",
}),

What do you think @danielweck ?

panaC added a commit that referenced this issue Dec 3, 2020
- filter alternate entrylinks in opds1
@panaC
Copy link
Member

panaC commented Dec 3, 2020

I was wrong on the borrow that opens in the browser.
The issue is here :

ln.type === ContentType.AtomXml) {

and will be probably fixed with
const contentType = parseContentType(ln.type);
if (contentType === ContentType.Opds2 ||
contentType === ContentType.Opds2Auth ||
contentType === ContentType.Opds2Pub ||
contentType === ContentType.AtomXml) {

@panaC
Copy link
Member

panaC commented Dec 3, 2020

tested with : https://github.com/vbessonov/opds-feed ,, it's works now !

@danielweck danielweck added this to Done in v1.6 Jan 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
v1.6
  
Done
5 participants