myenv subfolder is getting copied when serving the application

[mmabrouk]

Describe the bug
The way agenta serve works is that the whole folder is compressed and sent to the backend, which decompresses the folder and then create a container out of it.
The problem is that our tutorials ask the user to create a virutal environement in the folder they work in to test the app locally before serving it.
This means the whole myenv folder containg the python files (200M) is being compressed and sent each time to the backend which takes an eternity.

Expected behavior
We should find a way to solve this. The problem is the name of the folder can change. Some possible solutions are:

• Only copy .py, .yaml, .toml and .env files, for the rest the user needs to have a config file to explicitly allow other file format
• Only copy the content of the main folder. If the user wants to include subfolder, they need to explicitly allow it

We should research how other platforms do this (steamship, railway.app...)

In the long run, if we connect with git directly, then we make selecting the files a problem for the user.

This connect to #60

[aybruhm]

Hello @mmabrouk, I modified the add_variant function in the agenta-cli/agenta/cli/variant_commands.py module, a prototype for the above issue and would like to test it locally. How do I go about testing? I have searched the docs and could not find a way to do this.

P.S: When I run the command agenta variant serve, I'd like to view the logs (print statements) in the function.

[oussamajabnouni]

Hi,
I think every agenda app should include a .gitignore file. If it contains the .env file, users should be able to specify them on the dashboard. This would be a beneficial feature, especially if you plan to add git synchronization in the future.
Railway.app follows this approach and uses Nixpacks for building the app. It would be advantageous to use Nixpacks in Agenda, as it enables the development of agenda apps in a variety of languages.

[aybruhm]

@mmabrouk, here's how the prototype I built works (yet to be tested using the CLI): Allow the user to specify the environment folder name to exclude in the config.toml file that gets generated when they initialize their application. In a case where the user did not specify the folder to exclude, they get asked by the CLI when they run the command agenta variant serve, if they're running the application in a virtual environment, if the answer is yes, another question will be asked; prompting the user to input the name of the virtual environment folder.

With the above approach, users can define the name of the environment folder to exclude by editing the config.toml file before running the variant command (agenta variant serve). The CLI will read the specified folder name, exclude it during the build process, and then create the Docker image without that excluded folder.

[aybruhm]

Hi @mmabrouk,

I have considered using git directly and have come up with a proposed solution (in addition to the prototype) to exclude the environment folder during the build process. This solution will allow users to easily manage which folders they want to exclude without having to modify the configuration file, config.toml, explicitly.

Here's how the process would look step-by-step:

1. When a user initializes their application using the CLI (agenta init), it is automatically initialized with a .gitignore file.

2. The user can open the .gitignore file and add the name of the environment folder (e.g., "myenv" or any other name) to it. This ensures that the folder will be excluded from version control and not included in future commits.

3. When the user is ready to deploy the application, they can run the agenta variant serve command. During the build process, the .dockerignore file is automatically generated based on the contents of the .gitignore file.

4. Since the environment folder is listed in .gitignore, it will be excluded from the Docker image during the build, avoiding the unnecessary inclusion of large files.

What are your thoughts on this? I'd also like to get a second opinion on this solution, as it provides a simple and user-friendly way to handle the exclusion of the environment folder without requiring the user to edit the config.toml file directly.

[mmabrouk]

@aybruhm

How do I go about testing

Currently we do test manually using the examples in the folder examples. I would welcome any proposition for 1) a manual testing protocol 2) automatic testing in a later stage point

Here are my thoughts about the solutions:

Solution 1:

Asking the user when running agenta variant serve whether they are running a virtual env.
I think this would add additional friction to agenta variant serve. What I would instead would suggest is that we calculate the size of the folder, and if it is above a certain thresholder (let's say 5M), then we warn the user that their folder is large, note that could happen if they are running a virtual environement, ask them whether they would like to add folders to ignore before serving

Solution 2:

Using .gitignore for providing which folders to ignore
I like this idea a lot , we can add some defaults directly in the .gitignore (such as myenv)

[aybruhm]

@mmabrouk, thank you.

I'd suggest we go with solution 2, as it stands out to be the preferred approach in the industry due to its simplicity, familiarity and its alignment with best practices in Git version control.

By leveraging .gitignore, the project can provide an effective and user-friendly way to handle exclusion of folder without complicating the CLI or even disrupting the development workflow.

[aybruhm]

@mmabrouk, thank you.

I'd suggest we go with solution 2, as it stands out to be the preferred approach in the industry due to its simplicity, familiarity and its alignment with best practices in Git version control.

By leveraging .gitignore, the project can provide an effective and user-friendly way to handle exclusion of folder without complicating the CLI or even disrupting the development workflow.

In addition to solution 2, we can also borrow some of the ideas in solution 1, too.