Eliminating hard dependencies

[masayuki038]

As @justinclift mentions here, I think we need to move datasource / query_runner away from redash itself to resolve hard dependencies.

pyproject.toml has a group called all_ds. However, it is unlikely that you will use all Datasources in a production environment. There are also DBs like Trino that can connect and join multiple DBs. Therefore, with Redash, I think it is best to first focus on the relationship between one Redash and one Datasource.

My idea is as follows (very rough, because I'm not an expert on Python projects.)

1. monorepo (Redash, each datasource, frontend)
2. one docker container by one datasource

What do you think?

[georgialive]

Your idea of restructuring Redash into a monorepo architecture, where each data source and the frontend are modularized and potentially encapsulated in their own Docker containers, is quite innovative and could address several challenges in the current setup. Here are some thoughts and considerations on your approach:

Advantages of Monorepo and Containerization

1. Modularity: Separating each data source into its own module enhances modularity. This makes maintaining and updating specific data sources easier without impacting the entire system.

2. Dependency Management: By isolating data sources, you reduce the dependency load on the main Redash repository. Each data source can manage its dependencies independently, thus simplifying the overall dependency management.

3. Scalability and Flexibility: With Docker containers, scaling specific data sources becomes more manageable. Users can deploy only the containers they need, saving resources and reducing complexity.

4. Ease of Development and Testing: Developers can focus on specific modules or containers, making the development process more straightforward. Testing can also be more focused and efficient.

Challenges and Considerations

1. Complexity of Setup and Maintenance: While this architecture can simplify some aspects, it might introduce complexity in others, especially in terms of setup and maintenance. You'll need robust orchestration and management of multiple containers.

2. Overhead of Containerization: Running multiple containers might introduce performance overhead. It's crucial to ensure that the containerized approach doesn't significantly impact the performance or resource utilization.

3. Inter-Container Communication: Ensuring efficient and secure communication between different containers (and with the main Redash service) will be critical. You'll need to design a reliable and secure network architecture.

4. Documentation and User Guidance: This new architecture would require comprehensive documentation to guide users through setup, deployment, and troubleshooting. Clear and detailed documentation will be essential for user adoption.

5. Versioning and Compatibility: Handling versioning in a monorepo setup, especially with multiple independent modules, can be challenging. You'll need a strategy to ensure compatibility between different versions of Redash and its data source modules.

6. Community and Contributor Adaptation: Such a significant architectural change might require time for the community and contributors to adapt. It's important to consider how this transition will be managed.

Implementation Steps

1. Prototype: Start with a prototype involving a couple of data sources. This will help you identify and address potential issues early.

2. Feedback and Iteration: Seek feedback from the Redash community and iterate on your design based on their input.

3. Gradual Transition: Consider a gradual transition to the new architecture. This can help manage risks and allows users and contributors to adapt more easily.

4. Comprehensive Testing: Extensively test each part of the system, both individually and as a whole, to ensure stability and performance.

Your idea aligns well with modern software development practices emphasizing modularity, scalability, and maintainability. It's ambitious and would likely require considerable effort and community involvement, but it could significantly enhance Redash's architecture and usability.

[justinclift]

monorepo (Redash, each datasource, frontend)

Wouldn't it make more sense to have the Redash server (backend Python, frontend JS) be in the same repo, but have the data sources moved out? eg into their own repos

If the data sources are in the same monorepo as the rest of Redash, that doesn't seem like it will reduce the dependencies?

[masayuki038]

@justinclift @guidopetri

As a first thought I'd probably de-couple most query runners from the Redash repo though. That way people could develop and maintain their own ones without us needing to be involved.

Whether that's actually practical or a good idea though is another case. Might turn out to have obvious gotcha's that haven't occurred to me so far.

OK. So, as a first approach, how about we try separating the repositories? In any case, I don't think we'll know what the problem is until we try it.

We will create a repository for query_runner under getredash. Who should I reach out to to agree on this matter?

[guidopetri]

Sorry, I still have questions. What dependency problem are you referring to? All the query runner dependencies can install side by side, and I don't think we have any collisions on package versions. This also wouldn't eliminate e.g. the situation where a package is not maintained and we have to drop support for a data source as a result.

I think I am still unclear what the advantage is here.

[masayuki038]

@guidopetri Sure, good question.

I referred to these and started this thread. I'm wondering if it's quite difficult to accept the new query_runner now... How about that?

• Remove pygridgain so Trino data source shows up #6546
• Update Python dependencies to eliminate version conflicts #6122

If this problem is not a critical, please let me know 😅. I may have misunderstood the importance of the problem.

[guidopetri]

Ah, I see. That first PR, #6546 , was indeed a collision in versions, but one where what I said above applies - the package wasn't maintained correctly. The tzlocal version 2.1 was released in 2020, but pygridgain never updated from that and was stuck at that version.

The second PR #6122 was manually updating versions which is now managed by Poetry, so I don't think we have to worry about that situation.

I guess I'm just hesitant to spin off things into their own images and have one query runner image each because that means someone would have to run several docker containers, one per query runner that they support, and that could quickly get out of hand, especially with the memory overhead.

I'm also personally not a big fan of monorepo style repos, because I think they introduce far too much complexity in building and testing than they do in convenience, and I also don't think splitting query runners into their own repos is a good idea because that is additional overhead for developing - e.g. if we have to update a dependency across all query runners (base Python version?) or if someone wants to create a PR they have to set up each query runner repo locally.

[masayuki038]

@guidopetri Thanks for your reply. Understood.

I also don't would like to actively change this if possible. I just thought that if there was a dependency problem, I would have to deal with it.

I would like to close this discussion for now. If you have any opinions, please leave a comment. thank you.