Skip to content

Add support for Vagrant VMs in testing #20353

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
Show all changes
84 commits
Select commit Hold shift + click to select a range
5719504
vendor gstatus
NouemanKHAL Mar 25, 2025
58e04f5
fix mocks
NouemanKHAL Mar 31, 2025
ce7bb12
fix unit tests and mocks
NouemanKHAL Apr 3, 2025
9220006
remove gstatus from Dockerfile
NouemanKHAL Apr 7, 2025
2278322
add CheckVMLogs helper
NouemanKHAL May 22, 2025
579a08a
add support for vm_run context manager
NouemanKHAL May 22, 2025
4f70c80
test with glusterfs integration
NouemanKHAL May 22, 2025
9ea3d51
cleanup CheckVMLogs implementation
NouemanKHAL May 22, 2025
0344885
WIP: working VM setup + agent install , working ddev env agent commands
NouemanKHAL Jun 3, 2025
7f39a1d
working check command and config mounting
NouemanKHAL Jun 3, 2025
bbe7070
add support for vagrant in env_type in the plugin README
NouemanKHAL Jun 3, 2025
4c10969
fix glusterfs init config
NouemanKHAL Jun 3, 2025
4774f79
add gluster volumes -> E2E tests passing
NouemanKHAL Jun 3, 2025
f21b7ba
fix glusterfs version to 7
NouemanKHAL Jun 4, 2025
17364fa
revert plugin/pytest.py single quote -> double quote changes
NouemanKHAL Jun 4, 2025
6e7a8af
remove unnecessary formatting file changes
NouemanKHAL Jun 4, 2025
ac8527e
clean README changes
NouemanKHAL Jun 4, 2025
95b645f
add comment for CheckVMLogs condition
NouemanKHAL Jun 4, 2025
6083061
remove quote -> double quotes changes
NouemanKHAL Jun 4, 2025
a571131
conftest cleanup
NouemanKHAL Jun 4, 2025
6f87de2
add support for custom agent build via pipeline-id
NouemanKHAL Jun 12, 2025
efda745
wip: fix pip install command for local package
NouemanKHAL Jun 12, 2025
265d208
working --dev flag for local package installation
NouemanKHAL Jun 12, 2025
469d928
fix agent_env_vars keyword name and cleanup code
NouemanKHAL Jun 12, 2025
d05aebc
add support for custom memory and cpus for the vagrant VM
NouemanKHAL Jun 12, 2025
2a8ea78
fix exporting env vars + fix vm cpus and memory fields error
NouemanKHAL Jun 15, 2025
9bd48b1
add env metadata key
NouemanKHAL Jun 15, 2025
15af706
small cleanup
NouemanKHAL Jun 16, 2025
003ebac
refactor + fix dd_hostname handling
NouemanKHAL Jun 16, 2025
4f42406
more cleanup + fix agent restart cmd + use jinja for VagrantFile temp…
NouemanKHAL Jun 16, 2025
df351c2
remove checkVMLogs unused conditions
NouemanKHAL Jun 16, 2025
39bc26e
revert deleting the conditions.py file, only delete the new CheckVMLo…
NouemanKHAL Jun 17, 2025
132ee85
lint
NouemanKHAL Jun 23, 2025
f9535dd
fix version number committed by accident
NouemanKHAL Jun 25, 2025
870c184
wip
NouemanKHAL Jun 30, 2025
f637fe1
fix --dev local package installation, fix agent service restart, remo…
NouemanKHAL Jul 9, 2025
dd4858a
fix sudeoers config interface, fix sudoers bug in format and file nam…
NouemanKHAL Jul 9, 2025
739355f
Merge branch 'master' into noueman/glusterfs-vendor-gstatus
NouemanKHAL Jul 9, 2025
18a72a6
Merge branch 'noueman/glusterfs-vendor-gstatus' into noueman/add-supp…
NouemanKHAL Jul 9, 2025
f929b11
lint
NouemanKHAL Jul 9, 2025
7811862
ddev dep freeze
NouemanKHAL Jul 9, 2025
ebb6d9f
add paramiko dependency to ddev
NouemanKHAL Jul 9, 2025
b09b284
SKIP E2E tests on CI for glusterfs
NouemanKHAL Jul 9, 2025
3ec738d
fix SKIP E2E tests on CI for glusterfs
NouemanKHAL Jul 9, 2025
224c7e4
one more attempt to skip e2e tests on CI
NouemanKHAL Jul 9, 2025
113aba8
skip env provisioning when agent_type is vagrant on CI for the start …
NouemanKHAL Jul 9, 2025
a417392
fix running_on_ci check
NouemanKHAL Jul 9, 2025
0db2a02
remove paramiko
NouemanKHAL Jul 9, 2025
af575e9
add support for %HOST% template var and fix post_install_commands
NouemanKHAL Jul 12, 2025
c78d55f
always install glusterfs 11.1
NouemanKHAL Jul 13, 2025
a36c827
address review comments + refactor
NouemanKHAL Jul 13, 2025
9df460e
delete test_version_metadata from the e2e test
NouemanKHAL Jul 13, 2025
b06d000
improve logging, fix lint, fix commands output, use self.platform sub…
NouemanKHAL Jul 15, 2025
e7210de
more refactoring and code cleanup, and logs improvements
NouemanKHAL Jul 16, 2025
54211b1
refactor start method into smaller methods + cleanup
NouemanKHAL Jul 16, 2025
2200511
update AgentInterface to pass Application instead of Platform, and us…
NouemanKHAL Jul 16, 2025
6750042
remove unnecessary logging when raising exception
NouemanKHAL Jul 17, 2025
fd092c0
add VM hostname prefix to run_command debug logging
NouemanKHAL Jul 17, 2025
101ec82
add validation and proper error message around agent_build format
NouemanKHAL Jul 17, 2025
1291376
fix env reload command to only restart the agent service, not the ent…
NouemanKHAL Jul 17, 2025
f4f8b86
remove unused _show_logs method, and fix stderr logging level to be e…
NouemanKHAL Jul 17, 2025
d08884e
fix redundant log message of vagrant file generation, fix vagrant fil…
NouemanKHAL Jul 17, 2025
b35ca01
use ddev storage_dir to store the vagrant file
NouemanKHAL Jul 18, 2025
9978f56
fix exported_env_vars and propagate the right ddev config API keys
NouemanKHAL Jul 18, 2025
5b4c451
add test_vagrant.py + FIX windows bugs
NouemanKHAL Jul 18, 2025
e7e9b00
improve test names
NouemanKHAL Jul 18, 2025
9eb3cf5
use %HOST% variable in glusterfs start_commands
NouemanKHAL Jul 21, 2025
111dddd
improve logging messages
NouemanKHAL Jul 21, 2025
1431ac2
Apply suggestions from code review
NouemanKHAL Jul 21, 2025
dec2e7f
fix type hints
NouemanKHAL Jul 21, 2025
ef1ee14
remove unused constants
NouemanKHAL Jul 21, 2025
2f588ef
nit
NouemanKHAL Jul 21, 2025
8835dfb
apply suggestion to parametrize OS specific unit tests
NouemanKHAL Jul 21, 2025
0eb8f76
fix installing local packages on windows -> fix pip command
NouemanKHAL Jul 21, 2025
352e0ee
remove glusterfs changes from this branch
NouemanKHAL Jul 22, 2025
2f52be0
Merge branch 'master' into noueman/add-support-for-vms-in-testing
NouemanKHAL Jul 22, 2025
6103012
remove more glusterfs changes
NouemanKHAL Jul 22, 2025
07dd9d9
ddev lint
NouemanKHAL Jul 22, 2025
b392515
changelog
NouemanKHAL Jul 22, 2025
463de4a
Add Vagrant section in developer docs Test Framework
NouemanKHAL Jul 22, 2025
0fb956d
fix vagrant link
NouemanKHAL Jul 22, 2025
5ad27e3
revert dcd changes
NouemanKHAL Jul 24, 2025
30dc9da
Merge branch 'master' into noueman/add-support-for-vms-in-testing
NouemanKHAL Jul 24, 2025
14e6877
remove dcd changes
NouemanKHAL Jul 24, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ddev/changelog.d/20353.added
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add support for Vagrant VMs in testing
2 changes: 1 addition & 1 deletion ddev/pyproject.toml
Original file line number Diff line number Diff line change
@@ -133,4 +133,4 @@ ban-relative-imports = "parents"

[tool.ruff.lint.per-file-ignores]
#Tests can use assertions and relative imports
"**/tests/**/*" = ["I252"]
"**/tests/**/*" = ["I252"]
2 changes: 1 addition & 1 deletion ddev/src/ddev/cli/env/agent.py
Original file line number Diff line number Diff line change
@@ -38,7 +38,7 @@ def agent(app: Application, *, intg_name: str, environment: str, args: tuple[str

metadata = env_data.read_metadata()
agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, environment, metadata, env_data.config_file)
agent = get_agent_interface(agent_type)(app, integration, environment, metadata, env_data.config_file)

full_args = list(args)
trigger_run = False
2 changes: 1 addition & 1 deletion ddev/src/ddev/cli/env/reload.py
Original file line number Diff line number Diff line change
@@ -31,7 +31,7 @@ def reload_command(app: Application, *, intg_name: str, environment: str):

metadata = env_data.read_metadata()
agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, environment, metadata, env_data.config_file)
agent = get_agent_interface(agent_type)(app, integration, environment, metadata, env_data.config_file)

try:
agent.restart()
2 changes: 1 addition & 1 deletion ddev/src/ddev/cli/env/shell.py
Original file line number Diff line number Diff line change
@@ -33,7 +33,7 @@ def shell(app: Application, *, intg_name: str, environment: str):

metadata = env_data.read_metadata()
agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, environment, metadata, env_data.config_file)
agent = get_agent_interface(agent_type)(app, integration, environment, metadata, env_data.config_file)

try:
agent.enter_shell()
2 changes: 1 addition & 1 deletion ddev/src/ddev/cli/env/show.py
Original file line number Diff line number Diff line change
@@ -82,7 +82,7 @@ def show(app: Application, *, intg_name: str | None, environment: str | None, fo

metadata = env_data.read_metadata()
agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, environment, metadata, env_data.config_file)
agent = get_agent_interface(agent_type)(app, integration, environment, metadata, env_data.config_file)

app.display_pair('Agent type', agent_type)
app.display_pair('Agent ID', agent.get_id())
14 changes: 13 additions & 1 deletion ddev/src/ddev/cli/env/start.py
Original file line number Diff line number Diff line change
@@ -3,9 +3,17 @@
# Licensed under a 3-clause BSD style license (see LICENSE)
from __future__ import annotations

# Configure logging level from DDEV_LOG_LEVEL environment variable
import logging
import os
from typing import TYPE_CHECKING

import click
from datadog_checks.dev.ci import running_on_ci

log_level_str = os.environ.get('DDEV_LOG_LEVEL', 'INFO').upper()
log_level = getattr(logging, log_level_str, logging.INFO)
logging.basicConfig(level=log_level, format='%(levelname)s: %(message)s')

if TYPE_CHECKING:
from ddev.cli.application import Application
@@ -136,7 +144,11 @@ def start(
env_data.write_config(config)

agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, environment, metadata, env_data.config_file)

if agent_type == "vagrant" and running_on_ci():
app.abort(text="Vagrant is not supported on CI", code=0)

agent = get_agent_interface(agent_type)(app, integration, environment, metadata, env_data.config_file)

if not agent_build:
agent_build = (
2 changes: 1 addition & 1 deletion ddev/src/ddev/cli/env/stop.py
Original file line number Diff line number Diff line change
@@ -64,7 +64,7 @@ def stop(app: Application, *, intg_name: str, environment: str, ignore_state: bo
env_vars.update(metadata.get(E2EMetadata.ENV_VARS, {}))

agent_type = metadata.get(E2EMetadata.AGENT_TYPE, DEFAULT_AGENT_TYPE)
agent = get_agent_interface(agent_type)(app.platform, integration, env, metadata, env_data.config_file)
agent = get_agent_interface(agent_type)(app, integration, env, metadata, env_data.config_file)

try:
agent.stop()
35 changes: 35 additions & 0 deletions ddev/src/ddev/e2e/agent/Vagrantfile.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,35 @@
# -*- mode: ruby -*-
# vi: set ft=ruby :

$set_environment_variables = <<SCRIPT
tee "/etc/profile.d/myvars.sh" > "/dev/null" <<EOF

export LOCAL_IP=$(hostname -I | cut -d ' ' -f 1)
export DD_HOSTNAME=$(hostname)
{{ exported_env_vars_str }}

EOF
SCRIPT


Vagrant.configure("2") do |config|
config.vm.box = "{{ vagrant_box }}"
config.vm.box_version = "1.1"

{{ synced_folders_str | safe }}
config.vm.network "private_network", ip: "172.30.1.5"

config.vm.define "{{ vm_hostname }}" do |node|
node.vm.hostname = "{{ vm_hostname }}"

node.vm.provision "shell", inline: $set_environment_variables, run: "always"

node.vm.provision "shell", inline: <<-SHELL, run: "always"
sudo apt update
{{ agent_install_env_vars_str }} bash -c "$(curl -L https://install.datadoghq.com/scripts/install_script_agent7.sh)"
sudo service datadog-agent start && echo "Agent started successfully"
echo "VM {{ vm_hostname }} is ready"
SHELL
end

end
9 changes: 7 additions & 2 deletions ddev/src/ddev/e2e/agent/__init__.py
Original file line number Diff line number Diff line change
@@ -10,9 +10,14 @@


def get_agent_interface(agent_type: str) -> type[AgentInterface]:
if agent_type == 'docker':
if agent_type == "docker":
from ddev.e2e.agent.docker import DockerAgent

return DockerAgent

raise NotImplementedError(f'Unsupported Agent type: {agent_type}')
if agent_type == "vagrant":
from ddev.e2e.agent.vagrant import VagrantAgent

return VagrantAgent

raise NotImplementedError(f"Unsupported Agent type: {agent_type}")
12 changes: 12 additions & 0 deletions ddev/src/ddev/e2e/agent/constants.py
Original file line number Diff line number Diff line change
@@ -20,3 +20,15 @@ class AgentEnvVars:
SITE = 'DD_SITE'
TELEMETRY_ENABLED = 'DD_TELEMETRY_ENABLED'
URL = 'DD_DD_URL'


# Agent file paths (Linux)
LINUX_AGENT_BIN_PATH = "/opt/datadog-agent/bin/agent/agent"
LINUX_AGENT_PYTHON_PREFIX = "/opt/datadog-agent/embedded/bin/python"
LINUX_AGENT_CONF_DIR = "/etc/datadog-agent/conf.d"
LINUX_SUDOERS_FILE_PATH = "/etc/sudoers.d/dd-agent"

# Agent file paths (Windows)
WINDOWS_AGENT_BIN_PATH = "C:\\Program Files\\Datadog\\Datadog Agent\\bin\\agent.exe"
WINDOWS_AGENT_PYTHON_PREFIX = "C:\\Program Files\\Datadog\\Datadog Agent\\embedded"
WINDOWS_AGENT_CONF_DIR = "C:\\ProgramData\\Datadog\\conf.d"
11 changes: 8 additions & 3 deletions ddev/src/ddev/e2e/agent/interface.py
Original file line number Diff line number Diff line change
@@ -8,24 +8,29 @@
from typing import TYPE_CHECKING, Any

if TYPE_CHECKING:
from ddev.cli.application import Application
from ddev.integration.core import Integration
from ddev.utils.fs import Path
from ddev.utils.platform import Platform


class AgentInterface(ABC):
def __init__(
self, platform: Platform, integration: Integration, env: str, metadata: dict[str, Any], config_file: Path
self, app: Application, integration: Integration, env: str, metadata: dict[str, Any], config_file: Path
) -> None:
self.__platform = platform
self.__app = app
self.__integration = integration
self.__env = env
self.__metadata = metadata
self.__config_file = config_file

@property
def app(self) -> Application:
return self.__app

@property
def platform(self) -> Platform:
return self.__platform
return self.app.platform

@property
def integration(self) -> Integration:
Loading
Oops, something went wrong.
Loading
Oops, something went wrong.