Where should we put app/package data, and how do we manage that aspect?

[thorwhalen]

Sometimes we need our packages or apps to have some permanent storage to their disposal.

For example, right now; py2store makes a ~/.py2store_configs to manage store configs (and more)graze makes a ~/graze folder to put its stuff.
In all cases, the user is either warned or even asked for permission, because that's the kind of guy I am.

We could (and I'd like to) give the user the choice to specify a different location (in an environment variable, or a config file (but where? Ha!), perhaps even dynamically, in a user prompt, the first time it's needed.

Or perhaps we should just make the folder we need with out asking the user (like many other packages and apps seem to do).

Either way, where should the default location be? I'd like to have the same location for all our packages and apps. Makes it simpler.

But where?

I once found a huge article on this and thought I'd look through it one day and figure out a solution.

But today, I have chatGPT. After a bit of a conversation asking details about stuff, I finally came up with the right words to make it produce this:

import os

def get_app_data_folder():
    """
    Returns the full path of a directory suitable for storing application-specific data.

    On Windows, this is typically %APPDATA%.
    On macOS, this is typically ~/.config.
    On Linux, this is typically ~/.config.

    Returns:
        str: The full path of the app data folder.
    """
    if os.name == 'nt':
        # Windows
        app_data_folder = os.getenv('APPDATA')
    elif os.name == 'darwin':
        # macOS
        app_data_folder = os.path.expanduser('~/.config')
    else:
        # Linux/Unix
        app_data_folder = os.path.expanduser('~/.config')

    return app_data_folder

[thorwhalen]

Pushed this in this commit. But not closing until I include a tool for handling the other aspects (giving the user the ability to chose).

[thorwhalen]

I propose we use a {app_data_folder}/{package_name} protocol everywhere. Essentially, any package that needs a place for dynamic data should use {app_data_folder}/{package_name} by default.

[thorwhalen]

Here's a useful "package independent" piece of code to make and ensure the app data dir.

The verbose part of it could be replaced by a function (dol.ensure_dir accepts and can use callback functions as a verbose argument) that will ask the user for confirmation and/or if they want to accept the defaults, or even specify their own choice, using input or getpass.getpass.

import os
import i2, dol  # only used for `i2.get_app_data_folder` and `dol.ensure_dir`. Maybe we should centralize these?

def get_package_name():
    """Return package name (where this function is defined)"""
    return __name__.split('.')[0]  # TODO: See if this works in all cases where module is in highest level of package (if not, hardcode the package name

# get app data dir path and ensure it exists
pkg_name = get_package_name()
_root_app_data_dir = i2.get_app_data_folder()
app_data_dir = os.environ.get(
    f"{pkg_name.upper()}_APP_DATA_DIR", 
    os.path.join(_root_app_data_dir, pkg_name)
)
dol.ensure_dir(app_data_dir, verbose=f"Making app dir: {app_data_dir}")

[thorwhalen]

The function defined above returns the package name of the module where the function is defined.

To get the package name of the caller of a function, we can use the inspect module in Python. Specifically, we can use the inspect.stack() method to retrieve the current call stack and then access the package name of the caller from the frame object.

Here's an implementation of the function that returns the package name of the caller:

def get_caller_package_name():
    """Return package name of caller"""
    import inspect

    stack = inspect.stack()
    if len(stack) < 2:
        raise RuntimeError("No caller frame was found")
    try:
        caller_frame = stack[1][0]
        return inspect.getmodule(caller_frame).__name__.split('.')[0]
    except IndexError:
        raise IndexError("Couldn't find stack[1][0] of the stack")
    except Exception:
        raise RuntimeError("Couldn't find a module, or a name for it, in the caller frame")

In this implementation, we first retrieve the current call stack using inspect.stack(). The stack is a list of frame objects, where the first frame corresponds to the currently executing code and subsequent frames correspond to the call stack.

We check that the length of the stack is at least 2, because if there is only one frame, then there is no caller frame. In this case, we raise a RuntimeError.

We then retrieve the caller frame from the second element of the stack (stack[1]) and access the module object using inspect.getmodule(). Finally, we split the module name on the period character ('.') and return the first element of the resulting list, which is the package name.

Note that this implementation assumes that the caller is in a module that is part of a package. If the caller is in a module that is not part of a package (i.e., it is at the top level), the function will return None.

[thorwhalen]

If I write a package, or an app, where should I put the data, or the config files for that package or app? That is, where is the STANDARD place to put these things. I know it depends on the system (windows vs unix/mac), but does it also depend on whether it is configuration files or other kinds of data, or whether it's a package or an actual application. Give me a short answer, but also provide links of resources so I can get more information about the subject.

Bing Answer

The answer to your question depends on several factors, such as the type of data or configuration files, the operating system, the programming language, and the conventions of the package or app. There is no single standard place to put these files, but there are some general best practices and guidelines that you can follow.

One of the best practices is to separate the data and configuration files from the source code of the package or app. This makes it easier to maintain, update, and deploy the code without affecting the data or configuration. It also allows you to use different data or configuration files for different environments, such as development, testing, and production.

Another best practice is to use a consistent and logical structure for organizing the data and configuration files. For example, you can use subdirectories or prefixes to group related files together, or use descriptive and meaningful names for the files. You can also use standard formats for the files, such as JSON, XML, YAML, INI, etc., depending on the needs and preferences of your package or app.

Depending on the operating system, there are some common locations where you can store the data and configuration files. For example, on Windows, you can use the AppData folder for user-specific data and configuration files, or the ProgramData folder for system-wide data and configuration files. On Unix-like systems, such as Linux or Mac OS X, you can use the ~/.config folder for user-specific configuration files, or the /etc folder for system-wide configuration files. You can also use the ~/.local/share folder for user-specific data files, or the /usr/share folder for system-wide data files.

However, these locations are not mandatory, and you may need to consider other factors, such as security, accessibility, portability, and compatibility. For example, some packages or apps may need to store their data or configuration files in a specific location that is defined by their programming language or framework. Some packages or apps may also need to store their data or configuration files in a remote location, such as a database or a cloud service.

Therefore, there is no definitive answer to your question, but you can find more information about this topic from various sources. Here are some links that may be helpful:

• [5 Configuration Management Best Practices](^1^): This article discusses some general principles and patterns for managing configuration in applications.
• [Best practices for private config data and connection strings in configuration in ASP.NET and Azure](^2^): This article explains how to store sensitive data and connection strings securely in ASP.NET applications.
• [Best practices for deploying passwords and other sensitive data to ASP.NET and Azure](^3^): This article provides more details and examples on how to deploy passwords and other sensitive data to ASP.NET applications and Azure services.
• Where should I put my application's configuration settings?: This question on Stack Overflow has many answers that cover different scenarios and languages for storing configuration settings.
• Where should I store my application's data?: This question on Stack Overflow has many answers that cover different scenarios and platforms for storing application data.

Source: Conversation with Bing, 9/14/2023
(1) 5 Configuration Management Best Practices - InfoQ. https://www.infoq.com/articles/5-config-mgmt-best-practices/.
(2) Best practices for private config data and connection strings in .... https://www.hanselman.com/blog/best-practices-for-private-config-data-and-connection-strings-in-configuration-in-aspnet-and-azure.
(3) Best practices for deploying passwords and other sensitive data to ASP .... https://learn.microsoft.com/en-us/aspnet/identity/overview/features-api/best-practices-for-deploying-passwords-and-other-sensitive-data-to-aspnet-and-azure.