bug fix: Enforce power of 2 number of bytes per token

[le1nux]

What does this PR do?

Previously, the number of bytes per token was calculated by math.ceil(log_2(vocab_size)/8), leading to ranges between 1 and 4 bytes.
However, the dataset implementation only support 1, 2 and 4 bytes per token, as defined here

modalities/src/modalities/dataloader/dataset.py

Lines 202 to 206 in 0483362

	np_dtype_of_tokens_on_disk_from_bytes = {
	1: np.dtype(np.uint8).newbyteorder("<"),
	2: np.dtype(np.uint16).newbyteorder("<"),
	4: np.dtype(np.uint32).newbyteorder("<"),
	}

and

modalities/src/modalities/dataloader/dataset.py

Lines 233 to 234 in 0483362

	self._token_dtype_on_disk = self.np_dtype_of_tokens_on_disk_from_bytes[self._token_size_in_bytes]
	self._token_dtype_in_ram = self.type_converter_for_torch[self._token_size_in_bytes]

I added a switch case that maps to the respective byte sizes, when packing the data.

This adds some inefficiencies as a vobabulary size > 65536 already requires 4 bytes per token, effectively doubling the storage requirements.

Checklist before submitting final PR

• My PR is minimal and addresses one issue in isolation
• [] I have merged the latest version of the target branch into this feature branch
• I have reviewed my own code w.r.t. correct implementation, missing type hints, proper documentation, etc.
• I have run a sample config for model training
• I have checked that all tests run through (python tests/tests.py)
• I have updated the internal changelog (CHANGELOG_DEV.md)

[fromm-m]

LGTM

[flxst]

LGTM