Whether HDFS Dependencies Can Be Removed in the Future Development of Druid？

[Mrxiashu]

In the production environment, HDFS occupies some resources. If HDFS features (such as data ingestion) are not used, is HDFS optional in future design?

[jihoonson]

This has been my wish for a long time. Ideally, HDFS dependencies should be included only when you include Hadoop-related extensions. However, I don't think it could be done easily since it will require restructuring and refactorying lots of code. I don't think it could be done anytime soon.

[himanshug]

curious about what/how-much resources are being consumed (assuming you don't add hdfs-storage to extensions list anyways) ?

[pan3793]

Since Parallel Index has been introduced, is there a plan to decouple the indexing-hadoop and indexing-service? What I'm decouple is inverse the dependence of them and make indexing-hadoop optional, even move indexing-hadoop to extensions module resemble kafka-indexing-service

[github-actions]

This issue has been marked as stale due to 280 days of inactivity.
It will be closed in 4 weeks if no further activity occurs. If this issue is still
relevant, please simply write any comment. Even if closed, you can still revive the
issue at any time or discuss it on the dev@druid.apache.org list.
Thank you for your contributions.

[github-actions]

This issue has been closed due to lack of activity. If you think that
is incorrect, or the issue requires additional review, you can revive the issue at
any time.