Bot Opportunies

[xayhewalo]

Preface

The old Writing Bots Wiki used to have a list of "Bot Opportunities" that have been migrated here. The list migrated from the wiki will likely need further breakdown.

Contributors can edit this issue to track all bot-related clean up

Legend

• A: Authors
• E: Editions
• W: Works

Bot Clean up Opportunities

• Unicode [A, E, W]
  • 'mojibake' A~(c) style encoding errors. From multiple sources
    • Some from Amazon bulk imports
    • Some from external MARC records Some are lossy, others can be reconstructed. Important point to note: there are multiple styles that look similar, but require different translation to fix.
  • Combining diacritics. Some external MARC records have had subsequent correction, reimportation could catch many such cases Unicode: transcoding errors for \u0361 openlibrary#150
  • NFC Normalisation. Of titles, description, place, i.e. all text fields. Ensure all input mechanisms normalise correctly. UI + import paths. TEST.
• Spaces in OCAIDS [E] %20 space at end of Internet Archive ID is kept in "Read" url openlibrary#467
• URLs in OCAIDs [E]
  • not many?
• URLs in Amazon ids [E]
  • Needs checking. Some url ids appear to resolve correctly. Suspicion that some users are trying to insert affiliate links in OL ids. What is our policy on this? Should we be making more effort to ensure only Internet Archive affiliate links are used?
• Invalid ISBNs [E]
  • _10 and _13 Other ISBN fields; can they be used for anything?
    • repair if possible
    • normalise Fix Invalid ISBNs openlibrary-bots#43
    • remove if bad
    • test for reuse on multiple editions
• Invalid/Incorrect other ids [E, W]
  • LCCN LCCN incorrectly imported when containing alpha characters openlibrary#2851
  • OCLC
  • LibraryThing LibraryThingidentifiers are for works, not editions openlibrary#497
  • Sometimes ids are valid but link to a different book. How to resolve?
• Empty Titles [A, E, W]
  • Repair if possible
  • Delete if junk data
• Titles varied only by punctuation or whitespace (where author is the same) [E, W]
  • Remove trailing ".", ":", ";"
  • Reduce whitespaces to single sp
  • When one version has a single sp and another has no sp, change latter to single sp (e.g. Hitchhiker's guide --> Hitch hiker's guide)
  • When one version has embedded punctuation and another does not, change latter to match former (e.g. Hitch hikers guide --> Hitch-hiker's guide)
• Name Order [A]
  • The Open Library convention is natural name order as opposed to "Last, First" Is everyone happy with this standard? How should titles be handled? e.g. Sir , Lord, Lady, Mrs. There seems to be variation in the data for this style of name.
  • Also, how to handle Aliases? I notice the author_role type https://openlibrary.org/type/author_role has an 'as' property. Is this to allow for name variations on a particular book?
• Bad Alternate names for authors [A] Authors getting incorrect alternate names after merge openlibrary#498
• Orphaned Editions [E] Create Works for Work-less Editions openlibrary#153
• Works without Editions [W] Resolve (with redirects) Works w/ 0 Editions resulting from Workbot translation associations historical bug openlibrary#2727
  • WorkBot created editionless works openlibrary#44 Appears to be a merging task now. These editionless works look to be left over from moving editions (translations are a common example) to another work without converting the original work to a redirect. If there was some way to go back to the the original edition move and complete the work redirection? Perhaps the WorkBot log at https://github.com/internetarchive/openlibrary/blob/629f2afe8fbff9e8af50e00a57e756dff45a0d73/openlibrary/catalog/works/live.py#L438 might be helpful??
• Editions unlinked to Works [E]
  • Appears to occur mainly with apparently editionless works. How common is it? If rare, it should be fairly simple to address manually in librarian mode given a list of such editions.
• Authors without Work [A]
• Orphaned editions without Authors [A]
  • Where author is known to IA, from MARC records. Update the records from the original MARC. Many of the missing authors seem to be on books that have editors rather than straight authors. Is this a cause of the problem? How to correctly represent editors on Open Library works?
• Non-book Items Exclude dvds from non MARC imports openlibrary#2404

[BrittanyBunk]

@guyjeangilles I haven't created bots before (I'll look into it more), but I wish there's some description of how bots are created (like what databases they use - do they use the bulk data dump or the API? Idk how to access the API - as the website doesn't say).

[BrittanyBunk]

I think this relates to internetarchive/openlibrary#2902, because without data to be able to be easy to work with - it's going to be difficult to parse through to figure out that bots are needed and how to create bots to work with the data. Bots may also be able to work faster with less data too!

[xayhewalo]

@BrittanyBunk The API documentation could be better but you can most of it here: https://openlibrary.org/developers/api

Also the Bot Wiki is a little outdated but it's a good place to start. @hornc is a good resource as well.

[BrittanyBunk]

@guyjeangilles If that Bot Wiki could be added to the API documentation page on the OL, it would be an enormous help. I think that's the missing connection for explaining how to make bots! It's also nice that there's a new 'read' section too.

[BrittanyBunk]

I am going to list my bot opportunity ideas to keep in place here.

• More incorrect records beyond #1 openlibrary-librarians#10
• Bot suggestions openlibrary#2387

[LeadSongDog]

Another one: many author records have a trailing space in the name. These frequently are duplicates of records for the same author sans space. They should be corrected and, where appropriate, merged. See for example https://openlibrary.org/works/OL17769107W