Using opp_env: General discussion

[rhornig]

Feel free to share your ideas, use-cases regarding the opp_env tool.

[CafeKrem]

Hello,

I'm a beginner with OMNeT++, nix, and opp_env, and I have some suggestions/ideas for opp_env, especially for users with little or no experience with nix.

It would be useful if opp_env provided the following features:

1. Package Removal Command: A dedicated command or option to remove a package from the nix environment.
2. Garbage Collection: A built-in command to call nix-collect-garbage and clean up unnecessary dependencies.
3. Declarative Dependencies: A way to declare dependencies in a file, similar to requirements.txt in Python.
4. Dependency Listing: A command similar to pip freeze that outputs the currently installed dependencies.
5. Docker Image Generation: A command to generate a Docker image from the environment.
6. Improved Error Handling: More user-friendly error messages when attempting to install a package that is not available in nix. Currently, when trying to install an unsupported library, the error messages from nix can be unclear and difficult to interpret.

I assume that opp_env is built on nix and is designed to allow users to manage dependencies without requiring deep knowledge of nix. These features could make opp_env more intuitive and accessible.

I hope my use-cases are usefull ! And I'm sorry if I already cite points that are already implemented.

[avarga]

@CafeKrem Thank you for your suggestions! See comments below:

6. I assume you mean --add-extra-nix-packages. I'd also prefer a more user-friendly message, but I found no good and efficient/fast way to validate the package names against the Nix database, hence the current "solution" (i.e. let Nix fail). If you have an idea, please let me know.

7. Sounds reasonable, I'll discuss it with the team.

For 1-4, can you clarify them in the light of how the current dependency system works in opp_env? What currently happens is this:

Each version of each project has a JSON-style descriptor in the database; you can view this database with opp_env info --raw. Each entry describes the opp_env dependencies (required_projects key) and the Nix dependencies (nixos, nix_packages keys) of the project. The total list of Nix packages activated for the session (shell or run subcommand) are computed as the union of the nix_packages list of the project and of all its expanded dependencies, plus the extra Nix packages added to the workspace via --add-extra-nix-packages. Therefore, the Nix package dependencies are already declaratively defined. Not sure how "package removal" (1) would operate. About garbage collection (2), you are free to delete packages via Nix means if you think they won't be needed; opp_env will install them back if it still needs them for some project/workspace. BTW internally opp_env uses the nix develop command; you can make opp_env print what it does by specifying --debug in front of the shell / run subcommand.

[CafeKrem]

Hello,

Thank you both for the answers ! I appreciate the fast answers !

Sorry for the delays, I have a lot of work (I will respond to the point 6 for now). I will answer to other points tomorrow evening ( Sorry for that ).

I create a POC in bash that work well ( help with chatGPT ). That provide display the error with an additional message ( for example in case of wrong package we could add warning message like "The package does not exist, you can look at the available package on : https://search.nixos.org/packages?"

#!/bin/bash

# Define color codes
RED='\033[0;31m'    # Red for errors
YELLOW='\033[0;33m' # Yellow for warnings
GREEN='\033[0;32m'  # Green for success
NC='\033[0m'        # No Color (reset)

# Get package from command-line argument
PACKAGE=$1
if [[ -z "$PACKAGE" ]]; then
    echo -e "${RED}Error: No package specified.${NC}"
    exit 1
fi

# Check if package is already installed persistently
if nix profile list --extra-experimental-features nix-command --extra-experimental-features flakes | grep -q "$PACKAGE"; then
    echo -e "${GREEN}Package '$PACKAGE' is already installed persistently.${NC}"
else
    echo -e "${YELLOW}Installing package '$PACKAGE' persistently...${NC}"
    
    # Capture installation output in a variable
    INSTALL_OUTPUT=$(nix profile install nixpkgs#$PACKAGE --extra-experimental-features nix-command --extra-experimental-features flakes 2>&1)
    echo INSTALL = $INSTALL_OUTPUT
    # Check if the output contains "error" (case insensitive)
    if echo "$INSTALL_OUTPUT" | grep -qi "error"; then
        echo -e "${RED}Error encountered during installation:${NC}"
        echo -e "${RED}$INSTALL_OUTPUT${NC}"
        exit 1
    else
        echo -e "${GREEN}Package '$PACKAGE' installed successfully!${NC}"
    fi
fi

# Run opp_env shell
echo -e "${GREEN}Launching opp_env shell...${NC}"
exec opp_env shell --no-chdir

It's easier to write a POC than integrate into the opp_env code. I don't want to patronize anyone.

[rhornig]

2. sounds reasonable. I mean if we don't want to expose the user to the nix command directly, it is indeed a usecase to regularly run garbage collection so an opp_env gc subcommand is justified IMHO.

3. The project dependencies are indeed already defined declaratively as @avarga suggested. If you mean adding extra packages then again --add-extra-nix-packages can be used to add additional packages and those are indeed stored in a file in the .opp_env_workspace directory. You are free to edit that file. This is mostly used to define your own projects dependencies. If your project become part of the opp_env database, those dependencies will be incorporated in the main database.

4. Are you interested in nix dependencies (i.e. all the nix packages in the shell) or the modell dependendcies (i.e. simu5g x.y requires inet z.w and omnetpp a.b)? The latter is available in opp_env info command. The former can be extracted using opp_env info --raw, but I admit, you have to run this command for all the dependencies one by one and then build a union of that.

5. I was considering this feature (docker image gen) as it would be really useful to set up various CI images (or building images to be deployed on Windows Linux subsystem). I did a little experimenting with the nix docker builder machinery but was frightened by the way the image was generated (it created trillions of separate layers in the docker image which was definitely not for my liking). But again, I spent very little time with this and agree that this should be on the todo list.

As a side note, I have also a "dream" feature where (based on already existing opp_env database) the command could spit out a nix flake file, so you would be able to build a proper NIX package for the given project. That would allow the generation of nix packages for ALL the models in the database allowing binary build to be cached and pulled directly from nix cache without even building the model on the user's machine. This feature would be more useful for the end users who just want to use the model, but not develop. Right now we are more focused on developers who tend to modify and fiddle with the code.

We very much appreciate any feedback you can give.

[CafeKrem]

Thank you for your answers !

The project dependencies are indeed already defined declaratively as @avarga suggested. If you mean adding extra packages then again --add-extra-nix-packages can be used to add additional packages and those are indeed stored in a file in the .opp_env_workspace directory. You are free to edit that file. This is mostly used to define your own projects dependencies. If your project become part of the opp_env database, those dependencies will be incorporated in the main database.

OK I will look into it that directories, is there some docs about it ?

Are you interested in nix dependencies (i.e. all the nix packages in the shell) or the modell dependendcies (i.e. simu5g x.y requires inet z.w and omnetpp a.b)? The latter is available in opp_env info command. The former can be extracted using opp_env info --raw, but I admit, you have to run this command for all the dependencies one by one and then build a union of that.

For now, I'm only interested into using external dependency to use tool like pagmo, nanoflann. I made a custom simulator using vectorized operation, For my research work I don't need a packet-detail simulation ( But I will go deeper into packet level simulation for the futur works ).

I was considering this feature (docker image gen) as it would be really useful to set up various CI images (or building images to be deployed on Windows Linux subsystem). I did a little experimenting with the nix docker builder machinery but was frightened by the way the image was generated (it created trillions of separate layers in the docker image which was definitely not for my liking). But again, I spent very little time with this and agree that this should be on the todo list.

In my laboratory there is a project of building digital twin for electrical road infrastructure ( using veins ), by instantiating and configuring custom simulation/optimization problem. I will try to test dockerisation on nix on my free time.

As a side note, I have also a "dream" feature where (based on already existing opp_env database) the command could spit out a nix flake file, so you would be able to build a proper NIX package for the given project. That would allow the generation of nix packages for ALL the models in the database allowing binary build to be cached and pulled directly from nix cache without even building the model on the user's machine. This feature would be more useful for the end users who just want to use the model, but not develop. Right now we are more focused on developers who tend to modify and fiddle with the code

It will be fantastic !

If I add others ideas/remarks, I will tell you.

Sorry for my english.

[minhaj6]

Reading this discussion helped me clear my confusion more than the https://oppenv.omnetpp.org/ webpage. Things dicussed here about how things work, should make its way to the webpage/documentation section.