CSV import/export

[maarcingebala]

Implement CSV import/export for products.

Export

Proposed API:

type Mutation {
  exportProducts(scope: ExportScope, filter: ProductFilterInput, ids: [ID!]): Job!
}

enum ExportScope {
  ALL
  IDS
  FILTER
}

type Job {
  id: ID!
  status: String!
  createdAt: DateTime!
  url: String
}

Description

Mutation triggers a Celery task that takes given queryset of products and exports them to a file. As a result of the mutation, we return a Job object that informs about the status of the operation. Once the task is finished, Job will contain a link to download the CSV file (or a dashboard view with the report and download link). We will send this link in an email to the user who requested the operation.

Job objects could be used in the dashboard to render a list of scheduled jobs, query their status or access past exports, but we'll need to agree if this in the scope of this PR.

Parms:
My idea is to use scope, filter and ids to determine which products to export. filter would be of the same shape as filter used in the products query, so we can filter out some products first on the list and then export them. ids is to allow exporting only selected set of product ids. There is also the ability to export all products in the database. The scope param would determine which of these three export options to use.

Permissions: MANAGE_PRODUCTS (or specific permission only for import/export - needs to be discussed)

Import

type Mutation {
  importProducts(file: Upload!, updateExisting: Boolean!, updateStock: Boolean!): Job!
}

Description

Mutation passes the uploaded file to a task that reads and parses the file. For each row, we create necessary model instances, create connections between them and save them in the database. Once the task is finished we send an email to the staff user who requested the operation with a report about the number of successful/failed rows and/or errors that occurred.

Params:

• updateExisting - if true and there is a product/variant ID in CSV that already exists in the database, the import will override these products with data from CSV. If false, lines that refer to existing products will be skipped.
• updateStock - whether to update stock quantities with values from the CSV file. This is to guard against overriding the inventory accidentally with quantities that may be outdated in the file.

Permissions: same as exportProducts mutation.

Notes

• Rows in the CSV file will represent either products or variants. We will assume that the first line (after the header line) will represent a product and the following lines will represent its variants until the importer recognizes the next product.
• Product images will be exported as URLs; when importing we also expect URLs where we'll download the images from.
• The importer should be able to create new products, variants, attributes with values and images. For some other related objects, we'll assume that they exist in the database and we'll match them by IDs or slugs: warehouses, categories, collections, product types.
• The exact shape of CSV will be agreed on once we start working on this feature.

This is a draft and I'm open to suggestions, so we end up with API that we all agree with. The details of the shape of the proposed API is also to be adjusted if I missed something.

[maarcingebala]

Forgot to mention - we'll most probably use petl for CSV processing. Our team already has some experience with this lib plus it offers good memory efficiency.

[krzysztofwolski]

How should we handle the selection of exported fields? For example, cost_price should not be always exposed.
Can we extend export mutation for 'data shape' which would contain names of exported columns?

[ben-mandakh]

Any update on it? I am facing a problem to upload 100,000 products in 2 days? Is that possible to upload those as CSV directly to database?

[maarcingebala]

@benmandakh2609 No update so far, this task is scheduled to be worked on in our next core team sprint which starts at the end of January.

[itsgauravjain22]

I also need csv import export feature. Currently using woocommerce but can't switch to saleor due to this.

[jeremy-rutman]

any progress? I;d love to start importing products in order to use saleor but this is a blocker for us and I'm not keen on developing it myself if its in the pipeline in any case

[maarcingebala]

We had to pause development on CSV to focus on permission groups that were started earlier. I think as soon as we release them we'll get back to CSVs.

[AdamQuadmon]

Is there a way we can help with this?
How do you suggest to import thousands of products in the meanwhile?

[mgnanaprakash11]

When can we expect csv import/export feature?. Planning to start using saleor.

[thekennysong]

Anyone programmatically updating DB with python for updating products? I have a dictionary data type that I want to upload.

[aosorio1]

I'd love to see this feature as I'm right now also planning to move from Solidus to Saleor.

[aliasgartaksali]

Hey! Any progress on the tool? When can we expect it? Will be very helpful.

[robertology]

I started on an importer. Not CSV because I figured that would be a little limiting, but it's something. I no longer have the time nor drive to finish it, but it might be somewhat useful to someone.

Sorry to hijack this issue, but it's related and might help someone while waiting for something more official or complete.

https://github.com/robertology/saleor-import

[jeremy-rutman]

Any progress on this ?

[argenisleon]

Hi @robertology,

Do you have any idea about the performance? like products imported per minute?
I am using WordPress and we need to import 1000s of products but the most I can get is 10 products per minute which seem very slow

[robertology]

You're getting 10 per minute with my thing? I would expect much better than that, but I never ran it with 1000 items, so I don't really know how it would actually do.

But if you want to discuss more, probably best to open an issue on my repo to avoid bloating/derailing this one.

[timuric]

If you are looking for an intermediate solution, you can run a script that reads from a csv and creates products via graphql API, it is a reliable solution and it is fairly scalable, 10k products would not be a problem

[nux10]

Do you have a solution for Products import in Saleor ? (In python via GraphQL or directly in Saleor plugin's or application's)
Many thanks

[maarcingebala]

We started to add bulk mutations for data import, and this is the recommended approach now: https://docs.saleor.io/docs/3.x/developer/bulks/bulk-orders. CSV import won't be further developed in Saleor.