Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: csv loader #470

Merged
merged 20 commits into from
Sep 5, 2023
Merged

feat: csv loader #470

merged 20 commits into from
Sep 5, 2023

Conversation

cachho
Copy link
Contributor

@cachho cachho commented Aug 24, 2023

Description

Adds support for csv files.

  • local and remote files
  • detect delimiter
  • added tests

Each row is interpreted as a separate document, and stored as a comma separated string, where each row is prefixed with the header.

Name Age Occupation
Alice 28 Engineer
Bob 35 Doctor
Charlie 22 Student

Turns into

[
    {
        "content": "Name: Alice, Age: 28, Occupation: Engineer",
        "meta_data": {
            "url": "test.csv",
            "row": 1
        }
    },
    {
        "content": "Name: Bob, Age: 35, Occupation: Doctor",
        "meta_data": {
            "url": "test.csv",
            "row": 2
        }
    },
    {
        "content": "Name: Charlie, Age: 22, Occupation: Student",
        "meta_data": {
            "url": "test.csv",
            "row": 3
        }
    }
]

Fixes #87

Type of change

Please delete options that are not relevant.

  • New feature (non-breaking change which adds functionality)
  • Documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Please delete options that are not relevant.

  • Unit Test
  • Test Script (please provide)

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules
  • I have checked my code and corrected any misspellings

Maintainer Checklist

  • closes #xxxx (Replace xxxx with the GitHub issue number)
  • Made sure Checks passed

@cachho cachho mentioned this pull request Aug 24, 2023
14 tasks
@taranjeet taranjeet merged commit bd595f8 into mem0ai:main Sep 5, 2023
0 of 3 checks passed
taranjeet added a commit that referenced this pull request Sep 6, 2023
Co-authored-by: Taranjeet Singh <reachtotj@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature Request] Add CSV or Google Sheets support
2 participants