Skip to content

Change TokenLattice access level from public to internal #265

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Sep 25, 2025
Merged

Change TokenLattice access level from public to internal #265

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions Sources/Tokenizers/TokenLattice.swift
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
/// We could make it generic so TokenLatticeNode stores an opaque type, but it's overkill right now.
/// Based on https://github.com/huggingface/tokenizers/blob/b58227c7f1ccf8b73ee2268354336da56d91e492/tokenizers/src/models/unigram/lattice.rs#L137
/// and https://github.com/xenova/transformers.js/blob/b07336d8f7ff57453cc164cc68aead2a79cbd57e/src/utils/data-structures.js#L269C28-L269C28
public struct TokenLattice {
struct TokenLattice {
let sentence: String
let bosTokenId: Int
let eosTokenId: Int
Expand Down Expand Up @@ -40,7 +40,7 @@ public struct TokenLattice {
}
}

public extension TokenLattice {
extension TokenLattice {
/// Insert a new token into the node lattice.
///
/// - Parameters:
Expand Down Expand Up @@ -108,7 +108,7 @@ extension TokenLattice {
}
}

public extension TokenLattice {
extension TokenLattice {
var tokens: [String] {
viterbi().map { String(piece($0)) }
}
Expand Down