Repository navigation
PDF support added - #88
the-fallen wants to merge 5 commits into
Conversation
|
Nice, thanks for continuing with this! Just FYI, you know you can just reset a branch and Your choice of combining As for code details, some style needs cleaning (try |
|
@Treora As you mentioned, Readability mess is now in a seperate file, comments have been added in |
Treora
left a comment
There was a problem hiding this comment.
Good changes. Now I could understand the code better, and thereby find still more things to comment on. :)
|
|
||
| var totalContent = [] | ||
| var promises = [] | ||
| collectContent() |
There was a problem hiding this comment.
Function can just be inlined if it is only called and defined right here.
| callback(pageContent) | ||
| }) | ||
| }) | ||
| } |
There was a problem hiding this comment.
Why such a mix of callbacks and promises? And two arrays, for promises and text contents? Would something like this be more elegant? (probably contains mistakes)
const pageNumbers = range(pdf.pdfInfo.numPages) // import from lodash/fp/range
// (or come up with a cleaner for-loop replacement without range)
const pagesTextContentPs = pageNumbers.map(async i => {
const page = await pdf.getPage(i)
const pageTextPieces = await page.getTextContent()
const pageText = pageTextPieces.items.map(item => item.str).join(' ')
return pageText
})
const pagesTextContent = await Promise.all(pagesTextContentPs)
const totalDocumentText = pagesTextContent.join(' ')
There was a problem hiding this comment.
Also perhaps joining the pages with a few newlines instead of a space could still preserve some readability of the output. (I'm not sure what the 'items' are, I just named them textPieces above; somewhat meaningful names really helps readers understand the code)
| function getData(blob) { | ||
| return new Promise(function (resolve, reject) { | ||
| var fileReader = new FileReader() | ||
| fileReader.onload = async function (blob) { |
There was a problem hiding this comment.
This function feels too long to be nested. I would give it a name and put it above, and have it return the value, that is passed to resolve here.
| @@ -0,0 +1,34 @@ | |||
| import { getMetadata, metadataRules } from 'page-metadata-parser' | |||
| import extractPdfData from './extract-pdf-data' | |||
| import getText from './gettext' | |||
There was a problem hiding this comment.
I would try keep more informative names. extract-page-text was quite accurate, I think?
| resolve( | ||
| { | ||
| pageText: { bodyInnerText: totalContent }, | ||
| pageMetaData: JSON.parse(JSON.stringify(data.info, null, 2)), |
There was a problem hiding this comment.
stringify and parse? What does that achieve?
| var fileReader = new FileReader() | ||
| fileReader.onload = async function (blob) { | ||
| require('pdfjs-dist') | ||
| require('fs') |
There was a problem hiding this comment.
Do we need to import fs in the browser as well? Could be, but I thought it was a NodeJS thing.
| 'extractedMetadata.Title', | ||
| 'extractedMetadata.Author', | ||
| 'extractedMetadata.Subject', | ||
| 'extractedMetadata.Keywords', |
There was a problem hiding this comment.
Good idea to make these searchable. We may want to harmonise the metadata of html pages and pdfs; having title for one and Title for the other looks awful. The whole choice of fields and names is up for revision though, but we could do all this after this PR.
There was a problem hiding this comment.
Yes, it would be great if we could have same property keys.
| @@ -0,0 +1,58 @@ | |||
| // Returns a promise for an Object containing PDF Text and MetaData | |||
| function getData(blob) { | |||
There was a problem hiding this comment.
More meaningful name? We could actually expose just one extractPdfData, which accepts either a blob or a url, especially now url-to-blob adds just two lines of code.
export default async function extractPdfData({blob, url}) {
if (blob === undefined) {
response = await fetch(url)
blob = await ...
}
....
}
|
@Treora My bad, i didn't touch that part of oliver's code with callbacks, should've rewritten. Implemented now as you said with minor changes. Did all the other changes as you pointed out. |
|
Beautiful. I will test it out, if it works then it seems ready for merging (I will go through it once more and do some small comment rewording and style nitpicking). Do please let know if you copy parts of other people's code; I am quite conscious about copyright, trying to keep everything this repo in the public domain (should still declare that somewhere); see e.g. Unlicense. On that note, would you be willing to waive all your copyrights on your contributions? |
|
@Treora Yes, of course, i am willing. Also, no i haven't copied code from outside, nothing there to copy really, even the last traces of oliver's code are gone after rewriting pdf text extraction. |
|
I just tested this with a few more pdf files, seems to work so far. As discussed, the data model should be revised, and also problems may still pop up; especially performance on large documents could be an issue. But merging now anyway: we will discover and solve those problems as we go. Merged in d59ce46. |
|
And, of course, thanks for the effort! |
|
Happy to contribute! |
Sorry for the late commit, been really busy with college assignments lately.
Closed the last pull request and made this new one due to some issues.
As @Treora said, this commit takes cares of pdfjs-worker cleanup, doesn't load pdf twice and doesn't check if it's pdf twice.
The reason i think this is a better solution is if we do it in background, in case of a different module, we have to wait on an async chrome.tabs.query call everytime to get url and check if it's a pdf. Also, this saves alot of code replication.
Thanks alot for reviewing