Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

eScriptorium line-level ingest and editor #1504

Closed
9 tasks done
blms opened this issue Dec 8, 2023 · 1 comment
Closed
9 tasks done

eScriptorium line-level ingest and editor #1504

blms opened this issue Dec 8, 2023 · 1 comment
Assignees
@blms
Labels
🛠️ chore One-off task or update

Comments

@blms
Copy link
Contributor

blms commented Dec 8, 2023 •
edited by kseniaryzhova

testing notes (QA)

This is a test run, an initial line-level ingest of 200 files of automated transcriptions from eScriptorium.

List of eScriptorium files ingested, by PGPID 
PGPID_2926_MS-TS-00010-J-00018-00007_0.xml
PGPID_2926_MS-TS-00010-J-00018-00007_1.xml
PGPID_3182_MS-TS-00012-00057_0.xml
PGPID_3182_MS-TS-00012-00057_1.xml
PGPID_3182_MS-TS-AS-00150-00023_0.xml
PGPID_3182_MS-TS-AS-00150-00023_1.xml
PGPID_3455_MS-TS-00010-J-00005-00017_0.xml
PGPID_3455_MS-TS-00010-J-00005-00017_1.xml
PGPID_3455_MS-TS-00016-00022_0.xml
PGPID_3455_MS-TS-00016-00022_1.xml
PGPID_3907_MS-TS-AR-00043-00272_0.xml
PGPID_3907_MS-TS-AR-00043-00272_1.xml
PGPID_3907_MS-TS-AR-00047-00245_0.xml
PGPID_3907_MS-TS-AR-00047-00245_1.xml
PGPID_3907_MS-TS-AR-00049-00033_0.xml
PGPID_3907_MS-TS-AR-00049-00033_1.xml
PGPID_3907_MS-TS-G-00002-00060_0.xml
PGPID_3907_MS-TS-G-00002-00060_1.xml
PGPID_3907_MS-TS-G-00002-00060_2.xml
PGPID_3907_MS-TS-G-00002-00060_3.xml
PGPID_3907_MS-TS-G-00002-00060_4.xml
PGPID_3907_MS-TS-G-00002-00060_5.xml
PGPID_3907_MS-TS-G-00002-00060_6.xml
PGPID_3907_MS-TS-G-00002-00060_7.xml
PGPID_4715_MS-TS-00008-J-00021-00010_0.xml
PGPID_4715_MS-TS-00008-J-00021-00010_1.xml
PGPID_4722_MS-TS-00010-J-00013-00006_0.xml
PGPID_4722_MS-TS-00010-J-00013-00006_1.xml
PGPID_4727_MS-TS-00012-00337_0.xml
PGPID_4727_MS-TS-00012-00337_1.xml
PGPID_4728_MS-TS-00013-J-00007-00027_0.xml
PGPID_4728_MS-TS-00013-J-00007-00027_1.xml
PGPID_4737_MS-TS-AR-00007-00018_0.xml
PGPID_4737_MS-TS-AR-00007-00018_1.xml
PGPID_5355_MS-TS-00012-00176_0.xml
PGPID_5355_MS-TS-00012-00176_1.xml
PGPID_5355_MS-TS-00016-00146_0.xml
PGPID_5355_MS-TS-00016-00146_1.xml
PGPID_5370_MS-TS-NS-00184-00050_0.xml
PGPID_5370_MS-TS-NS-00184-00050_1.xml
PGPID_5370_MS-TS-NS-00184-00058_0.xml
PGPID_5370_MS-TS-NS-00184-00058_1.xml
PGPID_5370_MS-TS-NS-00184-00062_0.xml
PGPID_5370_MS-TS-NS-00184-00062_1.xml
PGPID_5370_MS-TS-NS-00184-00070_0.xml
PGPID_5370_MS-TS-NS-00184-00070_1.xml
PGPID_5370_MS-TS-NS-00184-00071_0.xml
PGPID_5370_MS-TS-NS-00184-00071_1.xml
PGPID_5370_MS-TS-NS-00184-00072_0.xml
PGPID_5370_MS-TS-NS-00184-00072_1.xml
PGPID_5370_MS-TS-NS-00184-00074_0.xml
PGPID_5370_MS-TS-NS-00184-00074_1.xml
PGPID_5370_MS-TS-NS-00184-00098_0.xml
PGPID_5370_MS-TS-NS-00184-00098_1.xml
PGPID_5427_MS-TS-00013-J-00007-00013_0.xml
PGPID_5427_MS-TS-00013-J-00007-00013_1.xml
PGPID_5427_MS-TS-K-00025-00252_0.xml
PGPID_5427_MS-TS-K-00025-00252_1.xml
PGPID_5427_MS-TS-NS-J-00005_0.xml
PGPID_5427_MS-TS-NS-J-00005_1.xml
PGPID_6052_MS-TS-MISC-00025-00130_0.xml
PGPID_6052_MS-TS-MISC-00025-00130_1.xml
PGPID_6053_MS-TS-MISC-00025-00133_0.xml
PGPID_6053_MS-TS-MISC-00025-00133_1.xml
PGPID_6058_MS-TS-MISC-00027-00004-00005_0.xml
PGPID_6058_MS-TS-MISC-00027-00004-00005_1.xml
PGPID_6059_MS-TS-MISC-00028-00207_0.xml
PGPID_6059_MS-TS-MISC-00028-00207_1.xml
PGPID_6060_MS-TS-MISC-00028-00235_0.xml
PGPID_6060_MS-TS-MISC-00028-00235_1.xml
PGPID_6061_MS-TS-MISC-00028-00240_0.xml
PGPID_6061_MS-TS-MISC-00028-00240_1.xml
PGPID_6062_MS-TS-MISC-00028-00246_0.xml
PGPID_6062_MS-TS-MISC-00028-00246_1.xml
PGPID_6063_MS-TS-MISC-00028-00250_0.xml
PGPID_6063_MS-TS-MISC-00028-00250_1.xml
PGPID_7050_MS-TS-00008-00029_0.xml
PGPID_7050_MS-TS-00008-00029_1.xml
PGPID_7051_MS-TS-00008-00030_0.xml
PGPID_7051_MS-TS-00008-00030_1.xml
PGPID_7052_MS-TS-00008-00032_0.xml
PGPID_7052_MS-TS-00008-00032_1.xml
PGPID_7053_MS-TS-00008-00033_0.xml
PGPID_7053_MS-TS-00008-00033_1.xml
PGPID_7054_MS-TS-00008-00038_0.xml
PGPID_7054_MS-TS-00008-00038_1.xml
PGPID_7056_MS-TS-00008-00041_0.xml
PGPID_7056_MS-TS-00008-00041_1.xml
PGPID_7057_MS-TS-00008-00042_0.xml
PGPID_7057_MS-TS-00008-00042_1.xml
PGPID_7058_MS-TS-00008-00048_0.xml
PGPID_7058_MS-TS-00008-00048_1.xml
PGPID_7065_MS-TS-00008-00057_0.xml
PGPID_7065_MS-TS-00008-00057_1.xml
PGPID_7066_MS-TS-00008-00058_0.xml
PGPID_7066_MS-TS-00008-00058_1.xml
PGPID_7067_MS-TS-00008-00061_0.xml
PGPID_7067_MS-TS-00008-00061_1.xml
PGPID_7068_MS-TS-00008-00063_0.xml
PGPID_7068_MS-TS-00008-00063_1.xml
PGPID_17325_MS-TS-AS-00145-00367_0.xml
PGPID_17325_MS-TS-AS-00145-00367_1.xml
PGPID_17328_MS-TS-AS-00145-00373_0.xml
PGPID_17328_MS-TS-AS-00145-00373_1.xml
PGPID_17329_MS-TS-AS-00145-00374_0.xml
PGPID_17329_MS-TS-AS-00145-00374_1.xml
PGPID_17330_MS-TS-AS-00145-00377_0.xml
PGPID_17330_MS-TS-AS-00145-00377_1.xml
PGPID_17331_MS-TS-AS-00145-00042_0.xml
PGPID_17331_MS-TS-AS-00145-00042_1.xml
PGPID_17333_MS-TS-AS-00145-00048_0.xml
PGPID_17333_MS-TS-AS-00145-00048_1.xml
PGPID_17334_MS-TS-AS-00145-00055_0.xml
PGPID_17334_MS-TS-AS-00145-00055_1.xml
PGPID_17335_MS-TS-AS-00145-00056_0.xml
PGPID_17335_MS-TS-AS-00145-00056_1.xml
PGPID_17336_MS-TS-AS-00145-00063_0.xml
PGPID_17336_MS-TS-AS-00145-00063_1.xml
PGPID_17337_MS-TS-AS-00145-00072_0.xml
PGPID_17337_MS-TS-AS-00145-00072_1.xml
PGPID_17338_MS-TS-AS-00145-00080_0.xml
PGPID_17338_MS-TS-AS-00145-00080_1.xml
PGPID_17339_MS-TS-AS-00145-00091_0.xml
PGPID_17339_MS-TS-AS-00145-00091_1.xml
PGPID_17340_MS-TS-AS-00145-00095_0.xml
PGPID_17340_MS-TS-AS-00145-00095_1.xml
PGPID_17341_MS-TS-AS-00146-00101_0.xml
PGPID_17341_MS-TS-AS-00146-00101_1.xml
PGPID_17342_MS-TS-AS-00146-00103_0.xml
PGPID_17342_MS-TS-AS-00146-00103_1.xml
PGPID_17343_MS-TS-AS-00146-00106_0.xml
PGPID_17343_MS-TS-AS-00146-00106_1.xml
PGPID_17344_MS-TS-AS-00146-00107_0.xml
PGPID_17344_MS-TS-AS-00146-00107_1.xml
PGPID_17345_MS-TS-AS-00146-00109_0.xml
PGPID_17345_MS-TS-AS-00146-00109_1.xml
PGPID_17346_MS-TS-AS-00146-00110_0.xml
PGPID_17346_MS-TS-AS-00146-00110_1.xml
PGPID_17347_MS-TS-AS-00146-00112_0.xml
PGPID_17347_MS-TS-AS-00146-00112_1.xml
PGPID_17348_MS-TS-AS-00146-00113_0.xml
PGPID_17348_MS-TS-AS-00146-00113_1.xml
PGPID_17349_MS-TS-AS-00146-00118_0.xml
PGPID_17349_MS-TS-AS-00146-00118_1.xml
PGPID_17350_MS-TS-AS-00146-00120_0.xml
PGPID_17350_MS-TS-AS-00146-00120_1.xml
PGPID_17351_MS-TS-AS-00146-00121_0.xml
PGPID_17351_MS-TS-AS-00146-00121_1.xml
PGPID_17352_MS-TS-AS-00146-00123_0.xml
PGPID_17352_MS-TS-AS-00146-00123_1.xml
PGPID_17353_MS-TS-AS-00146-00125_0.xml
PGPID_17353_MS-TS-AS-00146-00125_1.xml
PGPID_17354_MS-TS-AS-00146-00126_0.xml
PGPID_17354_MS-TS-AS-00146-00126_1.xml
PGPID_17355_MS-TS-AS-00146-00130_0.xml
PGPID_17355_MS-TS-AS-00146-00130_1.xml
PGPID_17356_MS-TS-AS-00146-00134_0.xml
PGPID_17356_MS-TS-AS-00146-00134_1.xml
PGPID_17357_MS-TS-AS-00146-00137_0.xml
PGPID_17357_MS-TS-AS-00146-00137_1.xml
PGPID_17358_MS-TS-AS-00146-00138_0.xml
PGPID_17358_MS-TS-AS-00146-00138_1.xml
PGPID_17360_MS-TS-AS-00146-00141_0.xml
PGPID_17360_MS-TS-AS-00146-00141_1.xml
PGPID_17361_MS-TS-AS-00146-00143_0.xml
PGPID_17361_MS-TS-AS-00146-00143_1.xml
PGPID_17362_MS-TS-AS-00146-00153_0.xml
PGPID_17362_MS-TS-AS-00146-00153_1.xml
PGPID_17363_MS-TS-AS-00146-00154_0.xml
PGPID_17363_MS-TS-AS-00146-00154_1.xml
PGPID_17364_MS-TS-AS-00146-00157_0.xml
PGPID_17364_MS-TS-AS-00146-00157_1.xml
PGPID_17365_MS-TS-AS-00146-00165_0.xml
PGPID_17365_MS-TS-AS-00146-00165_1.xml
PGPID_17366_MS-TS-AS-00146-00170_0.xml
PGPID_17366_MS-TS-AS-00146-00170_1.xml
PGPID_17367_MS-TS-AS-00146-00180_0.xml
PGPID_17367_MS-TS-AS-00146-00180_1.xml
PGPID_17368_MS-TS-AS-00146-00182_0.xml
PGPID_17368_MS-TS-AS-00146-00182_1.xml
PGPID_17369_MS-TS-AS-00146-00188_0.xml
PGPID_17369_MS-TS-AS-00146-00188_1.xml
PGPID_17370_MS-TS-AS-00146-00189_0.xml
PGPID_17370_MS-TS-AS-00146-00189_1.xml
PGPID_17371_MS-TS-AS-00146-00193_0.xml
PGPID_17371_MS-TS-AS-00146-00193_1.xml
PGPID_17372_MS-TS-AS-00146-00196_0.xml
PGPID_17372_MS-TS-AS-00146-00196_1.xml
PGPID_17373_MS-TS-AS-00146-00199_0.xml
PGPID_17373_MS-TS-AS-00146-00199_1.xml
PGPID_17374_MS-TS-AS-00146-00200_0.xml
PGPID_17374_MS-TS-AS-00146-00200_1.xml
PGPID_17375_MS-TS-AS-00146-00206_0.xml
PGPID_17375_MS-TS-AS-00146-00206_1.xml
PGPID_17376_MS-TS-AS-00146-00209_0.xml
PGPID_17376_MS-TS-AS-00146-00209_1.xml
PGPID_17377_MS-TS-AS-00146-00214_0.xml
PGPID_17377_MS-TS-AS-00146-00214_1.xml
PGPID_17378_MS-TS-AS-00146-00219_0.xml
PGPID_17378_MS-TS-AS-00146-00219_1.xml

On the QA site admin:

  • In the Log Entries section, you should see a bunch of new log entries from today 3/4/2024, including new annotations and new footnotes on the above documents
  • For any of the documents with PGPIDs in the list above, in the image/transcription viewer, you should see an option labeled "Edition: Machine-generated transcription (HTR for PGP model 1.0)" in the transcription section
    • Choosing this option should show the machine generated transcription, on the correct document and correct page
  • You should also see a link to edit the machine generated transcription
    • When you do this, you should see the new line-level annotation polygons and they should match up correctly with the lines of transcribed text. Use this method to spot check polygons.
    • You should be able to edit, save, and delete individual lines of text, and you should be able to edit and save block labels
    • You should not be able to drag/reorder lines at this point

On the QA public site:

  • The machine-generated transcriptions should be visible in the same way as on the admin site, on each document detail page
  • Machine-generated transcriptions should generally be indexed in the public site search, and thus full-text searchable—though we only index one transcription per document, so if there are other transcriptions on the document it is possible for the other(s) to take priority
@blms blms added the 🛠️ chore One-off task or update label Dec 8, 2023
@blms blms self-assigned this Dec 8, 2023
blms added a commit that referenced this issue Feb 7, 2024
@blms
Update escriptorium ingest script for line-level (#1504)
84bf1d7
blms added a commit that referenced this issue Feb 7, 2024
@blms
Update serialization to handle line-level annotations (#1504)
45d30d1
blms added a commit that referenced this issue Feb 7, 2024
@blms
Allow editing of line-level annotaitons (#1504)
3716fe9
@blms blms changed the title eScriptorium line-level ingest eScriptorium line-level ingest and editor Feb 7, 2024
blms added a commit that referenced this issue Feb 7, 2024
@blms
Fix bad line-level queryset condition (#1504)
623313e
blms added a commit that referenced this issue Feb 7, 2024
@blms
Fix unit tests and add some for line-level (#1504)
4e66980
@blms blms mentioned this issue Feb 7, 2024
Merged
blms added a commit that referenced this issue Feb 7, 2024
@blms
Improve test coverage for line-level annotations (#1504)
1f88a48
blms added a commit that referenced this issue Feb 19, 2024
@blms
Migrate logic from footnote to annotation (#1504)
adf95cc
blms added a commit that referenced this issue Feb 19, 2024
@blms
Fix bug in content_html (#1504)
37faaab
blms added a commit that referenced this issue Feb 19, 2024
@blms
Add logentry for line level ingest (#1504)
7e6f05b
blms added a commit that referenced this issue Feb 19, 2024
@blms
Improve documentation for block attribute (#1504)
ef80115
blms added a commit that referenced this issue Feb 19, 2024
@blms
Improve documentation of Annotation data model (#1504)
7eee853
blms added a commit that referenced this issue Mar 1, 2024
@blms
Use "machine-generated" for escr annotation export filename (#1504)
0434bdd
blms added a commit that referenced this issue Mar 1, 2024
@blms
Merge pull request #1537 from Princeton-CDH/feature/1504-line-model
5064150 
Allow ingest and editing of line-level annotations (#1504)
@blms blms added the 🗜️ awaiting testing Implemented and ready to be tested label Mar 4, 2024
@kseniaryzhova
Copy link

@blms amazing!!! Closing, it all works! Thanks!

@blms blms removed the 🗜️ awaiting testing Implemented and ready to be tested label Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@blms blms

Labels
🛠️ chore One-off task or update
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@blms @kseniaryzhova