Load file creation and modification date to metadata with DirectoryLoader

[PrimozGodec]

I was looking around the repository and web. I couldn't find the information if it is possible to load file creation and/or modification date together with a file with DirectoryLoader. Is it possible? How to do that?

Currently, I load only docx files, but I would also like to load some other types of documents in the future. My current code snippet is:

loader = DirectoryLoader(dir, glob="**/*.docx", show_progress=True, silent_errors=True)
docs = loader.load()

[dosubot[bot]]

🤖

Based on the current implementation of the DirectoryLoader class in the LangChain framework, it does not support loading file metadata such as creation and modification dates. The load_file method in the DirectoryLoader class only loads the content of the file into a Document object and does not extract or store any metadata about the file.

However, you can extend the DirectoryLoader class to include file metadata such as creation and modification dates. Here's how you can modify the load_file method to include the creation and modification dates:

import os
import time

def load_file(
    self, item: Path, path: Path, docs: List[Document], pbar: Optional[Any]
) -> None:
    """Load a file.

    Args:
        item: File path.
        path: Directory path.
        docs: List of documents to append to.
        pbar: Progress bar. Defaults to None.
    """
    if item.is_file():
        if _is_visible(item.relative_to(path)) or self.load_hidden:
            try:
                logger.debug(f"Processing file: {str(item)}")
                sub_docs = self.loader_cls(str(item), **self.loader_kwargs).load()

                # Get file metadata
                stat = os.stat(str(item))
                creation_date = time.ctime(stat.st_ctime)
                modification_date = time.ctime(stat.st_mtime)
                for doc in sub_docs:
                    doc.metadata['creation_date'] = creation_date
                    doc.metadata['modification_date'] = modification_date

                docs.extend(sub_docs)
            except Exception as e:
                if self.silent_errors:
                    logger.warning(f"Error loading file {str(item)}: {e}")
                else:
                    raise e
            finally:
                if pbar:
                    pbar.update(1)

In this modification, we use os.stat to get the file metadata, and time.ctime to convert the creation and modification times to a human-readable format. We then add these dates to the metadata of each document. Please note that this assumes that the Document class has a metadata attribute that is a dictionary. If this is not the case, you will need to modify the code accordingly.

You can find the original implementation of the load_file method in the DirectoryLoader class in the libs/langchain/langchain/document_loaders/directory.py file in the LangChain repository.

Sources

• libs/langchain/langchain/document_loaders/directory.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[PrimozGodec]

Bot, thank you for response. Based on your solution, I created a bit different code:

class DateDirectoryLoader(DirectoryLoader):
    def load_file(
        self, item: Path, path: Path, docs: List[Document], pbar: Optional[Any]
    ) -> None:
        prev_len = len(docs)
        super().load_file(item, path, docs, pbar)
        if len(docs) > prev_len:
            # if any file was loaded by super().load_file == no error loading
            stat = os.stat(str(item))
            creation_date = datetime.fromtimestamp(stat.st_ctime).isoformat()
            modification_date = datetime.fromtimestamp(stat.st_mtime).isoformat()
            for doc in docs[prev_len:]:
                doc.metadata['creation_date'] = creation_date
                doc.metadata['modification_date'] = modification_date

Key changes:

• I don't reimplement the whole method but only add a part that adds metadata. Loading is made by super call.
• I save the iso format of the date

I also have one important notice. stat.st_ctime is the creation date only in Windows and the metadata modification name on Unix. See for better solution here: https://stackoverflow.com/questions/237079/how-do-i-get-file-creation-and-modification-date-times

[Evil-Dr-Porkchop]

i try tour code but the returned Documents are still with the standard metadate with only source:

class EWICustomLoader2(DirectoryLoader):
    def load_file(
        self, item: Path, path: Path, docs: List[Document], pbar: Optional[Any]) -> None:
        prev_len = len(docs)
        print (prev_len)
        super().load_file(item, path, docs, pbar)
        if len(docs) > prev_len:
            #qui preparo i metadata...
            for doc in docs[prev_len:]:
                #print (doc)
                doc.metadata['test1'] = "test"
                doc.metadata['test2'] = "test"
        return docs

On main file i run the class:

loading_folder= "/opt/EAI_server/data/_loading"
loader_ewi = EWICustomLoader2(loading_folder, glob="**/*.txt")#, loader_cls=UnstructuredMarkdownLoader) #show_progress=True

docs = loader_ewi.load()
print (type(docs))
for doc in docs:
    print (type(doc))
    #print(doc.page_content)
    #print(doc.metadata)   
    print ("\n")

Do you have hint?