New Source: Magento

[cacan]

Tell us about the new integration you’d like to have

Magento1, Magento2 as source -> BigQuery as destination

Describe the context around this new integration

Reporting, ML, merging with analytics data
entities: products, categories, customers, orders

Describe the alternative you are considering or using

Singer

[sherifnada]

@cacan thanks for your request! Are you currently a Magento customer? It would help if we can mention you/your company when requesting access to a sandbox magento account. If we can get access to an account for integration testing, we'll be able to prioritize magento.

[cacan]

Hi @sherifnada,

thanks for your comment.

Most Magento installations are self-hosted, it has open source, and enterprise licence. For the most important entities [products, categories, orders, customers, ...], APIs are the same.

What exactly would you need? API endpoints and keys [+documentation]? Would that be sufficient?

Thank you...

[sherifnada]

@cacan Thanks for pointing out the open source version! I'll take a look. In general to build/maintain a connector, we need a way to run integration tests. For services which don't have a containerized version (e.g: Stripe API) this means creating an actual (potentially sandboxed) account, and hitting it from the integration tests. I'll look into the Magento OSS to see if it suffices for integration test needs.

What would be helpful to know from you is: To the extent you're able to share, what is the use case you have for moving the Magento data into BQ? The more detail the better as it helps us understand your use case and how to best support you.

[yevhenii-ldv]

Integration Vetting

Webhook-based? (no/partially/yes)

no

Available authentication modes (API key/Oauth/other)

API token

Creating an account

To work with Magento and Magento REST, you need to create your own domain, with which we can receive account data.
Unfortunately, I was unable to create a domain to work with Magento.

How to populate the account with data?

Sandbox

Available streams for sync

• Products
• Orders
• Customers
• Carts
• Magento_info
• Categories
• Stores

Integration supports incremental sync?

Yes(partially)

Other information/blockers

python client

[arhip11]

Hey @yevhenii-ldv and @sherifnada!
How do you see we go about it?
I see the following options:

• Custom deploy on AWS
• Use Sandbox (?)
• Find some Magento SaaS offerings

Please let me know if I should investigate anything.

[sherifnada]

@arhip11 is it possible to run this in Docker? that would be ideal for isolation of standard tests

[rjocoleman]

Magento 1 and Magento 2 are quite different.

Magento 1 is EOL, however there are some forks and LTS continuations and quite a number of merchants still out there.

Magento 2 is a rewrite with similar concepts and models but it's quite distinct.

Both use MySQL as data store. Both have REST APIs albeit with similar but different end points.
The python client mentioned in this thread targets Magento 1 mostly but might have some support for Magento 2.
The Magento 2 Rest API has Swagger specs built in so there are some codegen type clients possible.

Both Magento versions have Docker based dev type environments e.g. M2 https://github.com/bitnami/bitnami-docker-magento
M1: https://github.com/OpenMage/Docker-Setups (OpenMage is an LTS continuation)

Both versions come in various flavours - Open Source/Community and Enterprise (cloud or self-hosted). The difference between these are some additional plugins/overrides - the core feature set regarding orders, customers, shipments, products, quotes (carts), categories, stores are the same between OS and Enterprise in each version.

There is a rich ecosystem of third party extensions/module that can add both columns and tables.
Magento Marketplace (mentioned above with a link titled sandbox) is the official venue for selling these extensions/modules, I am not familiar with this link as I do not sell extensions there but I don't think it's a store sandbox.

There's is an official Magento 2 module that provides sample data: https://devdocs.magento.com/guides/v2.4/install-gde/install/sample-data-after-composer.html

There is certainly a way to spin up a Magento 2 dockerized container, probably a little overloaded with a webserver, php and mysql all in one and add the sample data to have reproducible builds for integration tests.

Personally my key interest is in Magento 1/OpenMage but I saw this issue so thought I'd try to help clear things up.

[cacan]

Glas to see this getting traction. Thanks to all involved, @rjocoleman covered it pretty well with OpenMage being the successor for M1. Although M2 had a rough start, it's getting more and more merchants to migrate from M1. Or other shopping carts. ;-)

Magento2 docker: you can also try some of these, they seem to be mature enough:
https://github.com/meanbee/docker-magento2
https://github.com/markshust/docker-magento

[sherifnada]

@rjocoleman and @cacan thanks for the amazing context! This definitely gives us a great idea to get started. Will keep you up to date with prioritization shortly.