Alignment of tokens for Grammer correction, where input is in-correct sentence while output/target is correct sentence

[akesh1235]

data

DatasetDict({
train: Dataset({
features: ['input', 'output'],
num_rows: 4500
})
test: Dataset({
features: ['input', 'output'],
num_rows: 500
})
})

# input (in-correct sentence)
data['train'][0]['input']
>> 'We are meet sunday 10am12pmET in Crown Heights Brooklyn New York'
# output (correct sentence)
data['train'][0]['output']
>> 'We meet Sundays 10am-12pmET in Crown Heights, Brooklyn, New York.'

I Want to align the output tokens with input

`# tokenize both inputs and targets
def tokenize_fn(batch):
    # tokenize the input sequence first
    # this populates input_ids, attention_mask, etc.

    tokenized_inputs = tokenizer(
    batch['input']
    )
    
    labels_batch = tokenizer.tokenize(batch['output']) # original targets

    aligned_labels_batch = []
    for i, labels in enumerate(labels_batch):
        word_ids = tokenized_inputs[i].word_ids()
        aligned_labels_batch.append(align_targets(labels, word_ids))  # align_targets is another user defined function which is been called here

    # recall: the 'target' must be stored in key called 'labels'
    tokenized_inputs['labels'] = aligned_labels_batch

    return tokenized_inputs`

data.map(
  tokenize_fn,
  batched=True,
  remove_columns=data['train'].column_names,
)

When this user defined function is mapped to every records of train and test batch am getting following error:

1. raise DatasetTransformationNotAllowedError(
3457 "Using .map in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."

2. TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

[mariosasko]

Hi! Can you provide the full error stack trace?