[Issue]: What input data formats are supported?

[yhshu]

Describe the issue

Hello, I see the documentation and the demo uses a long text file as input. Could there be any different input file formats? E.g., can we use multiple text files to store numerous documents?

Thank you!

Steps to reproduce

No response

GraphRAG Config Used

No response

Logs and screenshots

No response

Additional Information

• GraphRAG Version:
• Operating System:
• Python Version:
• Related Issues:

[KylinMountain]

only txt and csv, and csv need some extra config in settings.yaml

[jgbradley1]

As @KylinMountain stated, only txt and csv are supported file types at this time (specifically with utf-8 encoding). Multiple txt/csv files may be used.

[chongchongaikubao]

only txt and csv, and csv need some extra config in settings.yaml

请问还需要什么额外的config呢？

[KylinMountain]

@chongchongaikubao
不介意的话，可以看我的公众号文章解释了要如何配置。
GraphRAG如何配置处理csv文件

[nuocheng]

@chongchongaikubao 不介意的话，可以看我的公众号文章解释了要如何配置。 GraphRAG如何配置处理csv文件

你好，我按照你公众号进行csv进行配置，遇到了下面的问题
下面是我的配置和csv文件情况

input:
  type: file # or blob
  file_type: csv # or text
  base_dir: "input"
  file_encoding: utf-8
  file_pattern: ".*\\.csv$"
  source_column: "ArticleThemeCategory,ArticleType,Source"
  text_column: "ArticleContent,ArticleTitle,RelatedImageURL"
  timestamp_column: "ArticleDate"
  timestamp_format: "%Y-%m-%d"

这里的source_columns是不是类型列名，但是我的csv文件里存在多个类型列分别为ArticleThemeCategory,ArticleType,Source
text_column也是存在多个
这里的setting.yaml应该如何配置呢

[KylinMountain]

@nuocheng source_column 就类似于chunk的一段文本ID，或者说你csv中的第一列标明这一行的，起不到啥作用。
text_column是就是输入的文本的，但不能放多个列，你从哪里看到可以放多个列的哇

你从下面这个代码看，他并没有按照你想要的拆分啊，csv能支持的就几个列，你可能需要合并数据

data["text"] = data.apply(lambda x: x[csv_config.text_column], axis=1)

你可以考虑关注公众号，然后进群聊，能及时一点

[lalisalala]

Hi @KylinMountain , thank you for your help! So what if I have a csv with say 10 columns, like https://www.data.gov.uk/dataset/a2bd53f4-48b9-47b3-b994-1d8a380947c9/children-s-social-care-inspections-and-outcomes this one. How would I have to preprocess and change the settings.yaml in order to get the best outcome?

[KylinMountain]

@lalisalala
I can't access this file, can you add the column here? Basically, graphrag only accept source、 text 、title and timestamp column. So if you have content in several columns, you can try merge them into one with structed as text column and then prompt tune which will make it easier to extract the information in your column.

input:
  type: file # or blob
  file_type: csv # or text
  base_dir: "input"
  file_encoding: utf-8
  file_pattern: ".*\\.csv$"
  source_column: "maybe file name"
  text_column: "your content column"
  timestamp_column: "timestamp if you have"
  timestamp_format: "%Y-%m-%d"

[lalisalala]

@KylinMountain Thank you very much for your answer! I think I understood! Now in general, would you suggest that GraphRAG only succeeds in unstructured text data or is it also applicable for tabular structured data?

[akashAD98]

@KylinMountain Thank you very much for your answer! I think I understood! Now in general, would you suggest that GraphRAG only succeeds in unstructured text data or is it also applicable for tabular structured data?

yes I'm also looking for answer. i have csv with multiple columns name so will this work ?

[KylinMountain]

It is hard to say, since the entity extracted in graphrag need some description , and the description will be converted to embedding. Query like local query stage, it will compare the user query embedding with these descriptions. So if all of your entities extracted without description, I don't know how it works...maybe need you find a way to add some fixed description...