I dont know now how to read these failurs #74

Closed Answered by marcelroed

Darshith-845 asked this question in Help

Darshith-845
Jan 24, 2026

../tests/test_tokenizer.py::test_roundtrip_empty FAILED
../tests/test_tokenizer.py::test_empty_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_roundtrip_single_character FAILED
../tests/test_tokenizer.py::test_single_character_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_roundtrip_single_unicode_character FAILED
../tests/test_tokenizer.py::test_single_unicode_character_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_roundtrip_ascii_string FAILED
../tests/test_tokenizer.py::test_ascii_string_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_roundtrip_unicode_string FAILED
../tests/test_tokenizer.py::test_unicode_string_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_roundtrip_unicode_string_with_special_tokens FAILED
../tests/test_tokenizer.py::test_unicode_string_with_special_tokens_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_overlapping_special_tokens FAILED
../tests/test_tokenizer.py::test_address_roundtrip FAILED
../tests/test_tokenizer.py::test_address_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_german_roundtrip FAILED
../tests/test_tokenizer.py::test_german_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_tinystories_sample_roundtrip FAILED
../tests/test_tokenizer.py::test_tinystories_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_encode_special_token_trailing_newlines FAILED
../tests/test_tokenizer.py::test_encode_special_token_double_newline_non_whitespace FAILED
../tests/test_tokenizer.py::test_encode_iterable_tinystories_sample_roundtrip FAILED
../tests/test_tokenizer.py::test_encode_iterable_tinystories_matches_tiktoken FAILED
../tests/test_tokenizer.py::test_encode_iterable_memory_usage FAILED
../tests/test_tokenizer.py::test_encode_memory_usage XFAIL (Tokenizer.encode is expected to take...)

================================================= FAILURES =================================================
___________________________________________ test_roundtrip_empty ___________________________________________

def test_roundtrip_empty():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = ""

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:83: AttributeError
_______________________________________ test_empty_matches_tiktoken ________________________________________

def test_empty_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = ""

    reference_ids = reference_tokenizer.encode(test_string)

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:97: AttributeError
_____________________________________ test_roundtrip_single_character ______________________________________

def test_roundtrip_single_character():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "s"

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:113: AttributeError
__________________________________ test_single_character_matches_tiktoken __________________________________

def test_single_character_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "s"

    reference_ids = reference_tokenizer.encode(test_string)

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:127: AttributeError
_________________________________ test_roundtrip_single_unicode_character __________________________________

def test_roundtrip_single_unicode_character():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "🙃"

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:143: AttributeError
______________________________ test_single_unicode_character_matches_tiktoken ______________________________

def test_single_unicode_character_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "🙃"

    reference_ids = reference_tokenizer.encode(test_string)

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:157: AttributeError
_______________________________________ test_roundtrip_ascii_string ________________________________________

def test_roundtrip_ascii_string():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "Hello, how are you?"

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:170: AttributeError
____________________________________ test_ascii_string_matches_tiktoken ____________________________________

def test_ascii_string_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    test_string = "Hello, how are you?"

    reference_ids = reference_tokenizer.encode(test_string)

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:183: AttributeError
______________________________________ test_roundtrip_unicode_string _______________________________________

def test_roundtrip_unicode_string():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    test_string = "Héllò hôw are ü? 🙃"

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:199: AttributeError
___________________________________ test_unicode_string_matches_tiktoken ___________________________________

def test_unicode_string_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    test_string = "Héllò hôw are ü? 🙃"

    reference_ids = reference_tokenizer.encode(test_string)

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:212: AttributeError
____________________________ test_roundtrip_unicode_string_with_special_tokens _____________________________

def test_roundtrip_unicode_string_with_special_tokens():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    test_string = "Héllò hôw <|endoftext|><|endoftext|> are ü? 🙃<|endoftext|>"

  encoded_ids = tokenizer.encode(test_string)

                  ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:224: AttributeError
_________________________ test_unicode_string_with_special_tokens_matches_tiktoken _________________________

def test_unicode_string_with_special_tokens_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    test_string = "Héllò hôw <|endoftext|><|endoftext|> are ü? 🙃<|endoftext|>"

    reference_ids = reference_tokenizer.encode(test_string, allowed_special={"<|endoftext|>"})

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:241: AttributeError
_____________________________________ test_overlapping_special_tokens ______________________________________

def test_overlapping_special_tokens():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
        special_tokens=["<|endoftext|>", "<|endoftext|><|endoftext|>"],
    )
    test_string = "Hello, how <|endoftext|><|endoftext|> are you?<|endoftext|>"

  ids = tokenizer.encode(test_string)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:256: AttributeError
__________________________________________ test_address_roundtrip __________________________________________

def test_address_roundtrip():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    with open(FIXTURES_PATH / "address.txt") as f:
        corpus_contents = f.read()

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:273: AttributeError
______________________________________ test_address_matches_tiktoken _______________________________________

def test_address_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    corpus_path = FIXTURES_PATH / "address.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents)

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:287: AttributeError
__________________________________________ test_german_roundtrip ___________________________________________

def test_german_roundtrip():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    with open(FIXTURES_PATH / "german.txt") as f:
        corpus_contents = f.read()

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:302: AttributeError
_______________________________________ test_german_matches_tiktoken _______________________________________

def test_german_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    corpus_path = FIXTURES_PATH / "german.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents)

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:316: AttributeError
____________________________________ test_tinystories_sample_roundtrip _____________________________________

def test_tinystories_sample_roundtrip():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    with open(FIXTURES_PATH / "tinystories_sample.txt") as f:
        corpus_contents = f.read()

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:331: AttributeError
____________________________________ test_tinystories_matches_tiktoken _____________________________________

def test_tinystories_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    corpus_path = FIXTURES_PATH / "tinystories_sample.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents, allowed_special={"<|endoftext|>"})

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:344: AttributeError
_______________________________ test_encode_special_token_trailing_newlines ________________________________

def test_encode_special_token_trailing_newlines():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    corpus_path = FIXTURES_PATH / "special_token_trailing_newlines.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents, allowed_special={"<|endoftext|>"})

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:360: AttributeError
_________________________ test_encode_special_token_double_newline_non_whitespace __________________________

def test_encode_special_token_double_newline_non_whitespace():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    corpus_path = FIXTURES_PATH / "special_token_double_newlines_non_whitespace.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents, allowed_special={"<|endoftext|>"})

  ids = tokenizer.encode(corpus_contents)

          ^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode'

../tests/test_tokenizer.py:376: AttributeError
____________________________ test_encode_iterable_tinystories_sample_roundtrip _____________________________

def test_encode_iterable_tinystories_sample_roundtrip():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    all_ids = []
    with open(FIXTURES_PATH / "tinystories_sample.txt") as f:

      for _id in tokenizer.encode_iterable(f):

                   ^^^^^^^^^^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode_iterable'

../tests/test_tokenizer.py:390: AttributeError
____________________________ test_encode_iterable_tinystories_matches_tiktoken _____________________________

def test_encode_iterable_tinystories_matches_tiktoken():
    reference_tokenizer = tiktoken.get_encoding("gpt2")
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH, merges_path=MERGES_PATH, special_tokens=["<|endoftext|>"]
    )
    corpus_path = FIXTURES_PATH / "tinystories_sample.txt"
    with open(corpus_path) as f:
        corpus_contents = f.read()
    reference_ids = reference_tokenizer.encode(corpus_contents, allowed_special={"<|endoftext|>"})
    all_ids = []
    with open(FIXTURES_PATH / "tinystories_sample.txt") as f:

      for _id in tokenizer.encode_iterable(f):

                   ^^^^^^^^^^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode_iterable'

../tests/test_tokenizer.py:408: AttributeError
____________________________________ test_encode_iterable_memory_usage _____________________________________

@pytest.mark.skipif(
    not sys.platform.startswith("linux"),
    reason="rlimit support for non-linux systems is spotty.",
)
def test_encode_iterable_memory_usage():
    tokenizer = get_tokenizer_from_vocab_merges_path(
        vocab_path=VOCAB_PATH,
        merges_path=MERGES_PATH,
    )
    with open(FIXTURES_PATH / "tinystories_sample_5M.txt") as f:
        ids = []

      for _id in _encode_iterable(tokenizer, f):

../tests/test_tokenizer.py:427:

tokenizer = None
iterable = <_io.TextIOWrapper name='/home/darshith/assignment1-basics/tests/fixtures/tinystories_sample_5M.txt' mode='r' encoding='UTF-8'>

@memory_limit(int(1e6))
def _encode_iterable(tokenizer, iterable):
    """
    We place tokenizer.encode_iterable into a separate function so we can limit memory
    for just this function. We set the memory limit to 1MB.
    """

  yield from tokenizer.encode_iterable(iterable)

               ^^^^^^^^^^^^^^^^^^^^^^^^^

E AttributeError: 'NoneType' object has no attribute 'encode_iterable'

../tests/test_tokenizer.py:455: AttributeError
============================================= warnings summary =============================================
../tests/adapters.py:290
/home/darshith/assignment1-basics/tests/adapters.py:290: DeprecationWarning: invalid escape sequence '\T'
"""Given the weights of a Transformer language model and input indices,

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
========================================= short test summary info ==========================================
FAILED ../tests/test_tokenizer.py::test_roundtrip_empty - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_empty_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_roundtrip_single_character - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_single_character_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_roundtrip_single_unicode_character - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_single_unicode_character_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_roundtrip_ascii_string - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_ascii_string_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_roundtrip_unicode_string - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_unicode_string_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_roundtrip_unicode_string_with_special_tokens - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_unicode_string_with_special_tokens_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_overlapping_special_tokens - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_address_roundtrip - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_address_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_german_roundtrip - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_german_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_tinystories_sample_roundtrip - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_tinystories_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_encode_special_token_trailing_newlines - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_encode_special_token_double_newline_non_whitespace - AttributeError: 'NoneType' object has no attribute 'encode'
FAILED ../tests/test_tokenizer.py::test_encode_iterable_tinystories_sample_roundtrip - AttributeError: 'NoneType' object has no attribute 'encode_iterable'
FAILED ../tests/test_tokenizer.py::test_encode_iterable_tinystories_matches_tiktoken - AttributeError: 'NoneType' object has no attribute 'encode_iterable'
FAILED ../tests/test_tokenizer.py::test_encode_iterable_memory_usage - AttributeError: 'NoneType' object has no attribute 'encode_iterable'
================================ 24 failed, 1 xfailed, 1 warning in 13.54s =================================

These are the issues i face

Answered by marcelroed

This means your get_tokenizer_from_vocab_merges_path is returning None, or has no return statement. The error is telling you tokenizer is of type NoneType which doesn't have the method the test is expecting.

View full answer

Replies: 1 comment

marcelroed
Jan 26, 2026
Maintainer

This means your get_tokenizer_from_vocab_merges_path is returning None, or has no return statement. The error is telling you tokenizer is of type NoneType which doesn't have the method the test is expecting.

0 replies

Answer selected by marcelroed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment