Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generation of DjVu from JP2 files #18

Merged
merged 1 commit into from Feb 9, 2017

Conversation

Projects
None yet
3 participants
@samwilson
Copy link
Member

commented Jan 26, 2017

This adds a new system for producing DjVu files for IA items.

The biggest two parts of this are

  • a way to choose what the DjVu is created from (DjVu, JP2, or PDF);
  • and shifting the actually processing and uploading to a CLI.

@samwilson samwilson changed the title Generation of DjVu from JP2 files [WiP] Generation of DjVu from JP2 files Jan 26, 2017

@samwilson samwilson self-assigned this Jan 26, 2017

@samwilson samwilson force-pushed the djvu-generation branch from c223aa2 to afd70f5 Jan 27, 2017

@Tpt

This comment has been minimized.

Copy link
Collaborator

commented Jan 27, 2017

Thank you so much for working on it.

A deployment thing we should care about: the DjVu generation should probably be done on Tools Labs grid as it's probably a power intensive process. Doc: https://wikitech.wikimedia.org/wiki/Help:Tool_Labs/Grid

@samwilson

This comment has been minimized.

Copy link
Member Author

commented Jan 29, 2017

Yes, good point about using the Grid. My plan is to have DjVu-conversion jobs processed on the grid, launched every 1 minute or so. And to have a status list of current jobs on the homepage of the tool.

It'll have to save people's oauth tokens for the duration of the processing. I'll make sure this is very clear in the UI, and that the job.json files that store this info are only readable by the ia-upload user.

@samwilson samwilson force-pushed the djvu-generation branch 10 times, most recently from e24e52d to 37ec51f Jan 31, 2017

@samwilson

This comment has been minimized.

Copy link
Member Author

commented Feb 2, 2017

This is also being tracked at https://phabricator.wikimedia.org/T154538

@samwilson samwilson force-pushed the djvu-generation branch 8 times, most recently from 467ac5c to 9bd8c3f Feb 2, 2017

@kaldari

This comment has been minimized.

Copy link

commented Feb 7, 2017

According to the Phab ticket, this should be ready to merge. Please test and see if there are any last-minute kinks to work out.

@samwilson

This comment has been minimized.

Copy link
Member Author

commented Feb 7, 2017

This is also being tracked at https://phabricator.wikimedia.org/T157410

@samwilson samwilson changed the title [WiP] Generation of DjVu from JP2 files Generation of DjVu from JP2 files Feb 7, 2017

@samwilson samwilson force-pushed the djvu-generation branch 2 times, most recently from af28ac9 to 7260bd6 Feb 8, 2017

Generation of DjVu from JP2 or PDF files
Add a CLI system for processing the generation of DjVu files from
Internet Archive JP2 and PDF files.

Refs: #14 #13 #20

@samwilson samwilson force-pushed the djvu-generation branch from 7260bd6 to 93ba3bb Feb 8, 2017

@samwilson samwilson merged commit fdc94bd into master Feb 9, 2017

2 checks passed

continuous-integration/travis-ci/pr The Travis CI build passed
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@samwilson samwilson deleted the djvu-generation branch Feb 9, 2017

@samwilson

This comment has been minimized.

Copy link
Member Author

commented Feb 9, 2017

Deployed to production.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.