[pyos][design] Identification logic is weak, difficult to follow, and inconsistent.

[sneakers-the-rat]

Part of: pyOpenSci/software-submission#267 (comment)

If a DOI can be parsed from an item, then we look up the metadata from crossref and then return. that seems OK.

The rest of what happens in the identifier is a pretty big mystery. I would expect to just query all available metadata sources that have APIs that support that (e.g. sure you might not try and treat everything like a github URL, but if so far you haven't identified a work, you might as well try to check openaire).

Instead there is a ton of ad-hoc, in-place logic for determining where to check. Most of the time querying the sources is not even attempted because the logic is so specific, e.g. we have to have the zenodo DOI fragment present in the query string to check for it -

OneCite/onecite/pipeline.py

Line 531 in 12b1dea

zenodo_pattern = r'10\.5281/zenodo\.(\d+)'

, but that's a valid DOI so we should have already gotten the metadata for crossref! For some reason the figshare code is also in the zenodo method??? to check openaire we need to literally have the word "dissertation" or "phd thesis" in our query???

OneCite/onecite/pipeline.py

Lines 631 to 634 in 12b1dea

	thesis_keywords = [
	'phd thesis', 'ph.d. thesis', 'doctoral thesis', 'dissertation',
	'master thesis', "master's thesis", 'msc thesis', 'm.s. thesis'
	]

Why do we then try and guess the university pattern when we should be receiving that from the metadata provider???

OneCite/onecite/pipeline.py

Lines 671 to 674 in 12b1dea

	university_patterns = [
	r'(?:PhD|Ph\.D\.|Master|Doctoral|Dissertation).*?([A-Z][^.]*?University[^.]*?)\.?\s*$',
	r'([A-Z][^.]*?University[^.]*?)\.?\s*$',
	]

The whole identifier class needs to be restructured to have a sensible query logic, because as it stands now the identifier seems to pretty much always just query crossref, find a good enough match, and continue from there and the other sources are not attempted as much as I try to give examples that should be found elsewhere.

[sneakers-the-rat]

I am not even really sure where to start with the _fuzzy_search method which seems to be used most of the time for entries that don't have an explicit DOI in them -

OneCite/onecite/pipeline.py

Lines 1343 to 1653 in 12b1dea

	def _fuzzy_search(self, raw_entry: RawEntry,
	interactive_callback: Callable[[List[Dict]], int]) -> IdentifiedEntry:
	"""Perform fuzzy search"""
	query_string = raw_entry['query_string']
	# Check if it's a well-known paper first
	query_lower = query_string.lower()
	for key, paper_data in self.well_known_papers.items():
	if key in query_lower or fuzz.ratio(key, query_lower) > 85:
	self.logger.info(f"Entry {raw_entry['id']} matched well-known paper: {paper_data['title']}")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': None,
	'arxiv_id': paper_data.get('arxiv_id'),
	'url': paper_data.get('url'),
	'metadata': paper_data,
	'status': 'identified'
	}
	# Multi-source query with fallback
	candidates = []
	semantic_results = [] # Initialize to avoid UnboundLocalError
	# Check if query might be a PMID (PubMed ID)
	pmid_match = re.match(r'^(PMID:?\s*)?(\d{7,8})$', query_string.strip())
	if pmid_match:
	pmid = pmid_match.group(2)
	self.logger.info(f"Detected PubMed ID: {pmid}")
	pubmed_result = self._search_pubmed_by_id(pmid)
	if pubmed_result:
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': pubmed_result.get('doi'),
	'arxiv_id': None,
	'url': pubmed_result.get('url'),
	'metadata': pubmed_result,
	'status': 'identified'
	}
	# Check if it might be a book
	query_lower = query_string.lower()
	# === Strongest book indicators ===
	has_isbn = bool(re.search(r'isbn[:\s]*[\d\-xX]{10,17}', query_lower, re.IGNORECASE)) # ISBN-10 or ISBN-13
	has_editor = bool(re.search(r'\b(ed\.|eds\.|editor|editors|edited\s+by)\b', query_lower)) # Editor indicators
	# === Strong book indicators ===
	has_edition_number = bool(re.search(r'\d+(?:st|nd|rd|th)?\s+ed\.', query_lower)) # "2nd ed.", "3rd edition"
	has_press_publisher = bool(re.search(r'\b(mit|cambridge|oxford|springer|wiley|princeton)\s+press\b', query_lower))
	has_book_publisher = any(pub in query_lower for pub in ['wiley & sons', "o'reilly media", "o'reilly", 'addison-wesley', 'addison wesley'])
	has_publisher_location = bool(re.search(r'\b[A-Z][a-z]+,\s+(MA|NY|CA|UK):', query_string)) # "Cambridge, MA:" or "New York:"
	has_chapter = bool(re.search(r'\bchapter\s+\d+\b', query_lower)) # "Chapter 5"
	has_total_pages = bool(re.search(r'\b\d{2,4}\s+pp\.?\b', query_lower)) # "500 pp." (total pages)
	has_series = bool(re.search(r'\b(series|lecture notes)\b', query_lower)) # Book series
	# === Medium indicators ===
	mentions_edition = 'edition' in query_lower
	mentions_revised = any(word in query_lower for word in ['revised', 'updated', 'expanded'])
	# === Negative indicators (article/paper features) ===
	has_doi = bool(re.search(r'10\.\d{4,}/[^\s,}]+', query_string))
	has_journal_keywords = any(j in query_lower for j in ['journal of', 'proceedings of', 'conference on', 'transactions on', 'bulletin of', 'annals of'])
	has_volume_issue = bool(re.search(r'\b\d+\(\d+\)', query_string)) # Pattern like "83(8)" or "15(1)"
	has_article_pages = bool(re.search(r'\b\d{1,4}[-–]\d{1,4}\b', query_string)) and not has_total_pages # "123-456" but not "pp. 1-500"
	has_article_number = bool(re.search(r'\barticle\s+(number|no\.?)\b', query_lower))
	# === Decision logic with priority ===
	if has_isbn or has_editor:
	is_likely_book = True # ISBN or editor is definitive for books
	self.logger.info(f"Book indicators: ISBN={has_isbn}, Editor={has_editor}")
	elif has_doi or has_journal_keywords or (has_volume_issue and has_article_pages):
	is_likely_book = False # Definitely an article/paper
	elif has_edition_number or has_press_publisher or has_book_publisher or has_publisher_location or has_chapter or has_series:
	is_likely_book = True # Strong book signals
	elif mentions_edition and mentions_revised and not has_article_pages:
	is_likely_book = True # Revised edition mention
	else:
	is_likely_book = False # Default to article (safer)
	# === Route to the best API based on content type ===
	is_medical = any(keyword in query_lower for keyword in
	['health', 'medical', 'clinical', 'patient', 'disease', 'treatment',
	'therapy', 'diagnosis', 'biology', 'gene', 'protein', 'cell',
	'epidemiology', 'public health'])
	if is_likely_book:
	# === Book route ===
	self.logger.info("Likely a book, trying Google Books API first")
	books_results = self._search_google_books(query_string)
	if books_results and len(books_results) > 0:
	books_results[0]['is_primary_google_books_match'] = True
	candidates.extend(books_results)
	crossref_results = self._search_crossref(query_string)
	candidates.extend(crossref_results)
	elif is_medical:
	# === Medical route ===
	self.logger.info("Detected medical content, prioritizing PubMed")
	pubmed_results = self._search_pubmed(query_string)
	candidates.extend(pubmed_results)
	# Also try Crossref (many medical journals indexed there too)
	crossref_results = self._search_crossref(query_string)
	candidates.extend(crossref_results)
	# Semantic Scholar is good for medical CS papers (e.g., medical imaging AI)
	if any(cs_kw in query_lower for cs_kw in ['neural', 'deep learning', 'machine learning', 'ai']):
	semantic_results = self._search_semantic_scholar(query_string)
	candidates.extend(semantic_results)
	else:
	# === General academic paper route ===
	# Try Cross first (fastest and most reliable)
	crossref_results = self._search_crossref(query_string)
	candidates.extend(crossref_results)
	# If CrossRef results are good, also check Semantic Scholar for enrichment
	# (Semantic Scholar provides better metadata: citations, influential citations, etc.)
	if len(crossref_results) > 0:
	semantic_results = self._search_semantic_scholar(query_string)
	candidates.extend(semantic_results)
	self.logger.info(f"CrossRef found {len(crossref_results)} results, also querying Semantic Scholar for enrichment")
	# If CrossRef found nothing or poor results, use fallbacks
	if len(crossref_results) == 0 or (crossref_results and max(c.get('citations', 0) for c in crossref_results) < 10):
	self.logger.info("CrossRef results insufficient, trying fallbacks")
	# Try Semantic Scholar as primary fallback
	if not semantic_results: # If not already tried
	semantic_results = self._search_semantic_scholar(query_string)
	candidates.extend(semantic_results)
	# Google Scholar as last resort (if enabled)
	if self.use_google_scholar:
	self.logger.info("Google Scholar enabled, searching...")
	scholar_results = self._search_google_scholar(query_string)
	candidates.extend(scholar_results)
	else:
	self.logger.info("Google Scholar disabled by user, skipping")
	if not candidates:
	self.logger.warning(f"Entry {raw_entry['id']}: no candidate results found")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': None,
	'arxiv_id': None,
	'url': None,
	'metadata': {},
	'status': 'identification_failed'
	}
	scored_candidates = self._score_candidates(candidates, query_string)
	scored_candidates.sort(key=lambda x: x['match_score'], reverse=True)
	best_candidate = scored_candidates[0]
	# For likely books, strongly prefer Google Books results if available
	if not is_likely_book:
	is_likely_book = any(pub in query_string.lower() for pub in
	['wiley', 'pearson', "o'reilly", 'springer', 'press', 'publisher', 'edition', 'ed.'])
	if is_likely_book:
	# Find primary Google Books match (the first one returned by API)
	primary_google_books = next((c for c in scored_candidates if c.get('is_primary_google_books_match')), None)
	if primary_google_books and primary_google_books['match_score'] >= 35:
	# Use the primary (first) Google Books result - it's already sorted by relevance
	best_candidate = primary_google_books
	self.logger.info(f"Prioritizing PRIMARY Google Books result for book query (score: {best_candidate['match_score']})")
	self.logger.info(f"Selected book: {best_candidate.get('title', 'Unknown')} by {', '.join(best_candidate.get('authors', [])[:2])} ({best_candidate.get('year', 'N/A')})")
	else:
	# Fallback: use highest scoring Google Books result
	google_books_candidates = [c for c in scored_candidates if c.get('source') == 'google_books']
	if google_books_candidates and google_books_candidates[0]['match_score'] >= 40:
	best_candidate = google_books_candidates[0]
	self.logger.info(f"Prioritizing Google Books result for book query (score: {best_candidate['match_score']})")
	self.logger.info(f"Selected book: {best_candidate.get('title', 'Unknown')} by {', '.join(best_candidate.get('authors', [])[:2])} ({best_candidate.get('year', 'N/A')})")
	# Otherwise, prefer candidates with DOI when scores are close
	if best_candidate.get('source') != 'google_books':
	doi_candidates = [c for c in scored_candidates if c.get('doi')]
	if doi_candidates:
	best_doi_candidate = doi_candidates[0]
	if (not best_candidate.get('doi') or
	best_doi_candidate['match_score'] >= best_candidate['match_score'] - 5):
	best_candidate = best_doi_candidate
	# If best does not have DOI but looks strong, try title-only CrossRef lookup to resolve DOI
	if (not best_candidate.get('doi')) and best_candidate.get('title') and best_candidate.get('match_score', 0) >= 85:
	try:
	resolved = self._resolve_doi_via_crossref_title(best_candidate['title'], query_string)
	if resolved and resolved.get('doi'):
	best_candidate = resolved
	except Exception:
	pass
	# Decision logic
	if best_candidate['match_score'] >= 80:
	# High confidence: auto adopt
	if len(scored_candidates) == 1 or best_candidate['match_score'] - scored_candidates[1]['match_score'] > 10:
	self.logger.info(f"Entry {raw_entry['id']} high confidence match: {best_candidate.get('doi', 'no-doi')}")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': best_candidate.get('doi'),
	'arxiv_id': best_candidate.get('arxiv_id'),
	'url': best_candidate.get('url'),
	'metadata': best_candidate,
	'status': 'identified'
	}
	if 70 <= best_candidate['match_score'] < 80:
	# Medium confidence: trigger interactive mode
	top_candidates = scored_candidates[:5] # Top 5 candidates
	try:
	user_choice = interactive_callback(top_candidates)
	if 0 <= user_choice < len(top_candidates):
	chosen_candidate = top_candidates[user_choice]
	self.logger.info(f"Entry {raw_entry['id']} user selection: {chosen_candidate.get('doi', 'no-doi')}")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': chosen_candidate.get('doi'),
	'arxiv_id': chosen_candidate.get('arxiv_id'),
	'url': chosen_candidate.get('url'),
	'metadata': chosen_candidate,
	'status': 'identified'
	}
	else:
	# Non-interactive or user skipped: fallback to best candidate if sufficiently strong
	if best_candidate['match_score'] >= 75:
	self.logger.info(
	f"Entry {raw_entry['id']} fallback adopt best candidate (score={best_candidate['match_score']}): {best_candidate.get('doi', 'no-doi')}"
	)
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': best_candidate.get('doi'),
	'arxiv_id': best_candidate.get('arxiv_id'),
	'url': best_candidate.get('url'),
	'metadata': best_candidate,
	'status': 'identified'
	}
	self.logger.info(f"Entry {raw_entry['id']} user skipped")
	except Exception as e:
	self.logger.error(f"Interactive callback failed: {str(e)}")
	# Fallback in case interactive path is unavailable
	if best_candidate['match_score'] >= 75:
	self.logger.info(
	f"Entry {raw_entry['id']} fallback adopt best candidate after interactive error (score={best_candidate['match_score']}): {best_candidate.get('doi', 'no-doi')}"
	)
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': best_candidate.get('doi'),
	'arxiv_id': best_candidate.get('arxiv_id'),
	'url': best_candidate.get('url'),
	'metadata': best_candidate,
	'status': 'identified'
	}
	# Low confidence but if score is decent and has title, mark as identified
	# Lower threshold to avoid rejecting valid papers
	# For books, use even lower threshold (books often have less complete metadata)
	# Check if any keyword suggests this is a book
	is_best_candidate_book = (
	best_candidate.get('is_book') or
	best_candidate.get('type') in ['book', 'monograph', 'edited-book', 'reference-book'] or
	any(publisher in best_candidate.get('publisher', '').lower()
	for publisher in ['wiley', 'pearson', "o'reilly", 'springer'])
	)
	# For Google Books results, use even lower threshold (they're very accurate)
	if best_candidate.get('source') == 'google_books' and is_best_candidate_book:
	threshold = 35 # Very low threshold for Google Books + book detection
	elif is_best_candidate_book:
	threshold = 42 # Lower threshold for books
	else:
	threshold = 48 # Normal threshold for articles
	if best_candidate['match_score'] >= threshold and best_candidate.get('title'):
	self.logger.info(f"Entry {raw_entry['id']} adopting best candidate with score {best_candidate['match_score']} (type: {best_candidate.get('type', 'unknown')}, is_book: {is_best_candidate_book})")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': best_candidate.get('doi'),
	'arxiv_id': best_candidate.get('arxiv_id'),
	'url': best_candidate.get('url'),
	'metadata': best_candidate,
	'status': 'identified'
	}
	# Debug: Log the best candidate score for analysis
	self.logger.info(f"Entry {raw_entry['id']} best candidate score: {best_candidate.get('match_score', 0)}")
	if 'score_breakdown' in best_candidate:
	self.logger.info(f"Entry {raw_entry['id']} score breakdown: {best_candidate['score_breakdown']}")
	# Low confidence: mark as failed
	self.logger.warning(f"Entry {raw_entry['id']} low confidence match, marking as failed")
	return {
	'id': raw_entry['id'],
	'raw_text': raw_entry['raw_text'],
	'doi': None,
	'arxiv_id': None,
	'url': None,
	'metadata': {},
	'status': 'identification_failed'
	}

, the level of opacity and ad-hoc logic is stunning

[HzaCode]

Fixed in PR. Identifier routing simplified:

Core sources (always queried):

• CrossRef + Semantic Scholar for all text queries

Conditional specialized sources:

• ISBN/edition/publisher Google Books
• PMID pattern PubMed (early return for PMID-only queries)
• Strong biomedical cues PubMed (additive)
• Thesis keywords OpenAIRE/BASE
• Empty core results + enabled Google Scholar

Removed:

• is_medical branch (PubMed now additive, not exclusive)
• ~30 regex book detection simplified to 3 strong signals
• Google Books forced priority hack
• DOI forced priority hack

Added:

• DataCite fallback in _verify_doi_and_get_metadata for Zenodo/Figshare DOIs

[HzaCode]

Fixed in PR. Simplified routing: CrossRef+Semantic Scholar always queried; Google Books/PubMed/OpenAIRE conditionally appended. Removed is_medical branch and post-processing hacks.