Train wont start for custom dataset

[ArtBreguez]

I'm trying to train llama to customer support, i'm using a custom dataset that has this shape:

[
    {
        "query": "My order hasn't arrived yet.",
        "response": "We apologize for the inconvenience. Can you please provide your order number so we can investigate?"
    },
    {
        "query": "I received a damaged product.",
        "response": "We apologize for the inconvenience. Can you please provide a photo of the damaged product so we can assist you further?"
    },
    {
        "query": "I need to return an item.",
        "response": "Certainly. Please provide your order number and reason for return, and we will provide you with instructions on how to proceed."
    },
    {
        "query": "I want to change my shipping address.",
        "response": "No problem. Can you please provide your order number and the new shipping address you'd like to use?"
    },
]

I already tokenized, but when i run train.py the terminal freezes on:

tokens per iteration will be: 131,072
breaks down as: 4 grad accum steps * 1 processes * 128 batch size * 256 max seq len
Initializing a new model from scratch
num decayed parameter tensors: 43, with 15,187,968 parameters
num non-decayed parameter tensors: 13, with 3,744 parameters
using fused AdamW: False
Created a PretokDataset with rng seed 42

What could be the reason? When training with TinyStories it work fine.

[twobob]

Yeah. Im kinda in this space too. Figured it might be a me problem. still undecided.

Did try to poke around in exactly where the process thought it was by implementing the entire thing in colab and then stepping into the running process, but still did not figure it out beyond my "training has started" message that I injected fired...

have to revist

[madroidmaq]

@ArtBreguez you can try this #311 (comment) ，it‘s worked for me。