Emphasize AI in the first Apache release

[chenlica]

Our team has been developing a few powerful AI-related features in the past 1-2 years. For instance, we have an agent-based chatbot that can help users easily construct workflows using natural language and fix issues in workflows. We also have a few active efforts on using AI to improve various components of the system, and use the system to support AI-based analysis.

Given the increasing importance of AI in data science, it's important to emphasize these features and our effort in the direction. Therefore, I suggest the following:

1. Include an AI-based chatbot in the first Apache release.
2. Emphasize the system's position of using AI to support data science.

Thoughts?

[Yicong-Huang]

I support this idea in general. I think we have AI features that we should emphasis in this new era. It will take time to prepare the next release so it would be too late to bundle AI features in the next one.

However, I wonder if AI features would introduce more compelxcity on the license. I think we need to double check on that.

@chenlica if the discuss goes well, as this is release related, I think we need a apache vote on the email lit. of course, it will be based on the discussion result on this thread. The vote might be combined with the next RC vote. I am not sure about that though.

[Xiao-zhen-Liu]

I also generally support the idea, but for 1, I think we should be strategic about which AI features to include in the first release. We already have a basic AI chatbot feature merged in main (#4020) but the functionality is limited to read-only operations. This feature should be included in the first release. If we want to include more AI features, I think it makes more sense to let them be part of future releases instead of the first release, given the time constraint.

For 2, I wonder how we could emphasize it? In the documentations?

[Yicong-Huang]

For emphasize, I am thinking large here: we could even rebrand Texera entirely. Make AI the selling point of Texera. We used to use dataflow, GUI interface, collaborative, interactivity those as the keywords. We could reposition ourselves from the next release. That can be done on github, apache website, and also our outreach posts etc.

[aicam]

I also agree, I can think of even more sophisticated ways of using AI. For example, when user login for the first time, we only allow them to go through platform with the help of chatbot, in this context, chatbot not only control workflow, basically control the platform and do things on behalf of user.

[bobbai00]

I agree. And I shared the concern with @Yicong-Huang that it will introduce additional effort on the license compliance. But that effort should be manageble

[mengw15]

I generally support emphasizing AI in the first release. I agree that AI is becoming increasingly important for data science.

I support @Xiao-zhen-Liu’s comments that we should be strategic about which AI features to include, and avoid adding too much risk or delaying the release.

[Yicong-Huang]

Can we be more specific on the risk? Just want to understand what's different between "basic readonly AI features" and "advanced AI features" we are talking about, in terms of risks.

[mengw15]

I wasn’t referring only to the difference between “basic readonly AI features” and “advanced AI features.” My main concern is that many of the AI-related features are still not merged into the main branch yet, so I think we should take a more incremental approach.

Even if some of them are merged before the release, if developers have not had enough time to use and validate them in practice, there could still be potential bugs or integration issues. So my point is more about release readiness, rather than drawing a strict line between basic and advanced AI features.

[Yicong-Huang]

I see. That's very valid. On the side note, I think we should document AI features as "experimental", even on user-facing interfaces (i.e., on our UI). I think that can help relax the concern a bit. But I agree we should do it incrementally.

[chenlica]

For point 1), for sure we should try our best to test existing AI features to make sure they are reliable and usable. There are many AI features that could be developed in the future, and they shouldn't block us from introducing existing AI functionalities.

For point 2), I am thinking about rebranding the project in all public sites (GitHub, Apache Site, etc.).

[chenlica]

@zuozhiw Any thoughts from you?

[zuozhiw]

I strongly supports both 1 and 2. AI is more relevant than ever and Texera is well positioned. We should take the opportunity of release 1.0 and pivot completely into an AI first system.

In my view, this is not a change of mission for Texera, but a continuation of its original mission. Texera has always aimed to help less technical users do data science more easily. Today, AI is the most important new way to achieve that. Although AI is now everywhere, many of the latest and most effective practices are still mainly accessible to programmers. E.g. m, the best and greatest Claude code is primarily a terminal CLI. Texera is well positioned to close that gap and truly bring lots of values to our users.

I also want to add that “AI-first” does not mean Texera should become just another product with a big chat box as the primary focus. Texera already has a strong identity through its workflow UI, and I think that is a major advantage. Workflow UI visualization still has the advantages of being easy to understand. In the long term, workflow-based interaction can work very naturally with AI: AI can help explain workflows, generate operators or subflows, and assist users in refining pipelines, while the workflow itself remains core.

I saw there are concerns of how much AI we want to implement and if that will delay release. I agree we should be careful with release scope. But IMO releasing a version that no one cares about and fades into irrelevance, is far worse than delaying the release and make the release more AI ready. I think we should at least implement the readonly helping users to explain workflows Xiaozhen mentioned, and support using AI to generate single operators.

[chenlica]

Thanks to all who provided your input. Glad to see the support. We will include these two changes as part of the vote for the first Apache release.

[chenlica]

I sent a new positioning text to the dev email list: https://lists.apache.org/thread/r1gcrlqqvgln6222zkv7vqml9n6owp98