Wrap FileNotFound exceptions in the finetuning dataloader and convert_text_to_mds
#1246
Conversation
| @@ -683,6 +686,8 @@ def build_collate_fn( | |||
| 'timeout': 0, | |||
| }) | |||
|
|
|||
| mosaicml_logger = maybe_create_mosaicml_logger() | |||
There was a problem hiding this comment.
idk if this is necessary, given that this exception was already logged somewhere. is this caught and logged somewhere else (similar to convert_text_to_mds)?
|
@jjanezhang @angel-ruiz7 This is awesome! I was actually just seeing a lot of FileNotFound errors last week. Are there any plans to handle FileNotFound errors that are surfaced when downloading checkpoints? I saw some errors last week due to a bad custom weights path. (To be clear, this is out of scope for this PR, just wanted to get your thoughts 😄) |
| @@ -354,3 +356,26 @@ class RunTimeoutError(InternalError): | |||
| def __init__(self, timeout: int) -> None: | |||
| message = f'Run timed out after {timeout} seconds.' | |||
| super().__init__(message, timeout=timeout) | |||
|
|
|||
|
|
|||
| class DatasetMissingFileError(UserError): | |||
There was a problem hiding this comment.
Is this error supposed to cover both cases where a shard file is missing AND the index.json metadata file is missing? From the PR description, it sounds like this error is only for a missing/badly specified index.json file. So maybe it's more appropriate to call this DatasetMissingIndexError instead
| ) -> None: | ||
| message = "Could not find the file '{file_name}' with any of the supported extensions: " | ||
| message += ', '.join(supported_extensions) + '.' | ||
| message += ' Please check your train / eval data and try again.' |
There was a problem hiding this comment.
| message += ' Please check your train / eval data and try again.' | |
| message += ' Please check your train / eval dataset configuration and try again.' |
Not sure how this will be surfaced or if users will have access to this info, but this is more specific at least
| def __init__( | ||
| self, | ||
| file_name: str, | ||
| supported_extensions: Any, # Run into type error when using List[str] |
There was a problem hiding this comment.
what's the type error that you get? We do use List[str] in other exceptions, see here
llm-foundry/llmfoundry/utils/exceptions.py
Line 227 in 9fbfccc
List[str] here.
|
@angel-ruiz7 Are you still working on this? |
Creates two exceptions:
DatasetMissingFileError--> Tells the user that a dataset file could not be found during a failure in the finetuning dataloader wheresplit/pathwere not defined properlyDatasetInvalidFolderError--> Tells the user that aninput_folderis invalid and while trying to list objects inconvert_text_to_mds.pymapi exceptions implemented in https://github.com/databricks-mosaic/mcloud/pull/4088