Add integration tests for flatpak module

[oolongbrothers]

SUMMARY

This adds integration tests for the flatpak module.

I promised these when the module PR was merged. There is a companion pull request open that accomplishes the same for the companion flatpak_remote module: #42315

I did not include these in any of the ci groups, since there a a number of external dependencies (additional information below) and thus can not guarantee the highest level of reliability.

ISSUE TYPE

• Bugfix Pull Request

In the sense that not having integration tests could be regarded as a bug. But seriously, none of the categories really fit this.

COMPONENT NAME

flatpak

ANSIBLE VERSION

ansible 2.7.0.dev0 (flatpak_integration_tests 703e643517) last updated 2018/07/01 14:30:28 (GMT +200)
  config file = None
  configured module search path = [u'/home/[redacted]/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /home/[redacted]/Code/ansible/lib/ansible
  executable location = /home/[redacted]/Code/ansible/bin/ansible
  python version = 2.7.12 (default, Dec  4 2017, 14:50:18) [GCC 5.4.0 20160609]

ADDITIONAL INFORMATION

The tests:

• Are destructive and should thus be run using docker or vagrant. Only the docker method was verified by me.
• Require privileged mode when run with docker, because flatpak requires access to container control features and docker does not support container nesting in a less-privileged way. So these tests are not fully contained when running in docker. If this concerns you, you should try vagrant.
• Can run on the following distributions : Ubuntu 16.04+, Fedora 24+
• Rely on the external flathub repository to test the functionality
• Rely on the official flatpak PPA repo to set up flatpak on Ubuntu

When developing, I use a local docker image based on ansible/ansible:fedora25, that caches the flatpak installation for improved turnaround time:

$ ansible-test integration -v --docker fedora25-flatpak --docker-privileged --docker-no-pull --allow-unsupported flatpak
WARNING: Skipping docker pull for "ansible/ansible:fedora25-flatpak". Image may be out-of-date.
Creating a compressed tar archive of path: .
Resulting archive is 29899310 bytes.
Run command: docker run --detach --volume /sys/fs/cgroup:/sys/fs/cgroup:ro --privileged=true --volume /var/run/docker.sock:/var/run/docker.sock ansible/ansible:fedora25-flatpak
Run command: docker exec -i 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 dd of=/root/docker.sh bs=65536
Run command: docker exec 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 /bin/bash /root/docker.sh
Run command: docker exec -i 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 dd of=/root/ansible.tgz bs=65536
Run command: docker exec 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 mkdir /root/ansible
Run command: docker exec 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 tar oxzf /root/ansible.tgz -C /root/ansible
Run command: docker exec 638075f903b4124a654abe3e58ab8522df7b69010b51891a5273f36bf79d1258 /root/ansible/test/runner/test.py integration -v --docker-no-pull flatpak --metadata metadata-OOmhhp.json --truncate 238 --color yes --requirem ...
Run command: /usr/bin/python setup.py egg_info
Run command: /usr/bin/python -m pip.__main__ install --disable-pip-version-check -c test/runner/requirements/constraints.txt -r test/runner/requirements/integration.txt
Ignoring cryptography: markers u"python_version < '2.7'" don't match your environment
Ignoring astroid: markers u"python_version >= '3.5'" don't match your environment
Ignoring pylint: markers u"python_version >= '3.5'" don't match your environment
Ignoring sphinx: markers u"python_version < '2.7'" don't match your environment
Ignoring wheel: markers u"python_version < '2.7'" don't match your environment
Ignoring yamllint: markers u"python_version < '2.7'" don't match your environment
Ignoring paramiko: markers u"python_version < '2.7'" don't match your environment
Ignoring pytest: markers u"python_version < '2.7'" don't match your environment
Ignoring ordereddict: markers u"python_version < '2.7'" don't match your environment
Requirement already satisfied (use --upgrade to upgrade): cryptography in /usr/lib64/python2.7/site-packages (from -r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): jinja2 in /usr/lib/python2.7/site-packages (from -r test/runner/requirements/integration.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): junit-xml in /usr/lib/python2.7/site-packages (from -r test/runner/requirements/integration.txt (line 3))
Requirement already satisfied (use --upgrade to upgrade): paramiko in /usr/lib/python2.7/site-packages (from -r test/runner/requirements/integration.txt (line 5))
Requirement already satisfied (use --upgrade to upgrade): pyyaml in /usr/lib64/python2.7/site-packages (from -r test/runner/requirements/integration.txt (line 6))
Requirement already satisfied (use --upgrade to upgrade): idna<2.6 in /usr/lib/python2.7/site-packages (from -c test/runner/requirements/constraints.txt (line 13))
Requirement already satisfied (use --upgrade to upgrade): pyasn1>=0.1.8 in /usr/lib/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/lib/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): setuptools>=11.3 in /usr/lib/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/lib/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/lib/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.4.1 in /usr/lib64/python2.7/site-packages (from cryptography->-r test/runner/requirements/integration.txt (line 1))
Requirement already satisfied (use --upgrade to upgrade): MarkupSafe in /usr/lib64/python2.7/site-packages (from jinja2->-r test/runner/requirements/integration.txt (line 2))
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/lib/python2.7/site-packages (from cffi>=1.4.1->cryptography->-r test/runner/requirements/integration.txt (line 1))
Running flatpak integration test role
Run command: ansible-playbook flatpak-pPRT3w.yml -i inventory -e @integration_config.yml -v
Using /root/ansible/test/integration/integration.cfg as config file

PLAY [testhost] ****************************************************************

TASK [Gathering Facts] *********************************************************
ok: [testhost]

TASK [flatpak : Install flatpak on Fedora] *************************************
ok: [testhost] => {"changed": false, "msg": "Nothing to do"}

TASK [flatpak : Activate flatpak ppa on Ubuntu] ********************************
skipping: [testhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [flatpak : Install flatpak package on Ubuntu] *****************************
skipping: [testhost] => {"changed": false, "skip_reason": "Conditional result was False"}

TASK [flatpak : Enable flathub for user] ***************************************
changed: [testhost] => {"changed": true, "command": "/usr/bin/flatpak remote-add --user flathub https://dl.flathub.org/repo/flathub.flatpakrepo", "rc": 0, "stderr": "GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.\n", "stderr_lines": ["GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications."], "stdout": "", "stdout_lines": []}

TASK [flatpak : Enable flathub for system] *************************************
changed: [testhost] => {"changed": true, "command": "/usr/bin/flatpak remote-add --system flathub https://dl.flathub.org/repo/flathub.flatpakrepo", "rc": 0, "stderr": "GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications.\n", "stderr_lines": ["GLib-GIO-Message: Using the 'memory' GSettings backend.  Your settings will not be saved or shared with other applications."], "stdout": "", "stdout_lines": []}

TASK [flatpak : Test executable override] **************************************
fatal: [testhost]: FAILED! => {"changed": false, "msg": "Executable 'nothing-that-exists' was not found on the system."}
...ignoring

TASK [flatpak : Verify executable override test result] ************************
ok: [testhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [flatpak : Test addition (check mode)] ************************************
changed: [testhost] => {"changed": true, "command": "/usr/bin/flatpak install --system -y flathub org.gnome.Calculator", "rc": 0, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [flatpak : Verify addition test result (check mode)] **********************
ok: [testhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

TASK [flatpak : Test non-existent idempotency of installation (check mode)] ****
changed: [testhost] => {"changed": true, "command": "/usr/bin/flatpak install --system -y flathub org.gnome.Calculator", "rc": 0, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}

TASK [flatpak : Verify non-existent idempotency of installation test result (check mode)] ***
ok: [testhost] => {
    "changed": false, 
    "msg": "All assertions passed"
}

...

[ansibot]

cc @JayKayy
click here for bot help

[ansibot]

The test ansible-test sanity --test integration-aliases [explain] failed with 1 error:

test/integration/targets/flatpak/aliases:0:0: missing alias `posix/ci/group[1-3]` or `unsupported`

click here for bot help

[oolongbrothers]

In my view, getting this merged is a prerequisite for any further development on the module and thus blocks issue #41996

[gundalow]

Hi,
I'm randomly selecting some PRs to see how it was for you, with the aim of making it easier for future contributors. As a new contributor I'm particularly interested as you don't have any prior knowledge of the Ansible process.

1. Generally speaking, how was the process for you?
2. Did it make sense what to do to progress the PR?
3. Did you read any of the developer (dev_guide) or contributor documentation?
4. What in the documentation was useful or confusing, missing or could be improved
5. Did you know you could run ansible-test locally?
6. What areas would you be interested in contributing to?

ansible/community#353

[oolongbrothers]

Hello @gundalow ! Thanks for this initiative. I am happy to share my experience.

Actually, this PR is not my first. It rather is my third. I noticed the new_contributor tag on my first PR, on the second it was gone but on the third is was back. Maybe has something to do with that I changed e-mailaddresses a few months back, no change to the GitHub nick though.

My work on Ansible has mostly been to contribute to getting the new flatpak and flatpak_remote modules merged and trying to maintain them. I am struggling with the latter though.

1. In theory the process is straight forward and easy. Everybody has been welcoming and helpful.

2. The big problem for me in all of my three Ansible PRs has been to get attention to them after getting an initial review. Everything really depends on the initial reviewer. If they don't come back to the PR, hardly anybody else will. To get attention, I have (sparingly) pinged the reviewers, commented on and referenced my PRs in other PRs and issues. This sometimes eventually helps, but it takes months. I had to bring up my first PR in an IRC meeting to get any attention to it and get it merged after we had worked on it for months and had resolved all feedback.

3. and 4. Yes, I have read what I could find. Documentation about the process was informative, reasonably scoped and relatively easy to find, even though I remember that I had to actively look for documentation on testing and I wished I had known more about how to write and run integration tests. Information on the IRC meetings was also scarce.

4. Yes. I did use this from my first PR. However, I did not understand how to run the integration tests to make sure that things work in both Python 2 and 3.

5. I am trying to be maintainer for the flatpak and flatpak_remote modules I co-authored. I am having a hard time though. Pull requests take months to get merged and in the mean time other pull requests get merged that break the module. There is duplication of effort where a new PR gets merged that fixes a bug that I have already submitted a PR for that got stalled. So far every such fix has introduced regressions in the module. I am running after these making new PRs that then get stalled. One big reason is that integration tests currently are not run in CI because I fear that they would be too brittle. But I don't get time to work on this or the list-as-input-to-package-module feature because I run after these regressions.
   Also, I don't always get notified about bugs that people file on the module.
   This all makes it very hard to feel good about the work I put into Ansible. I am certain that everybody tries to do their best, it rather is a structural question. At this point, however, I don't really know how to make the development process work for me.
   I might not be the quickest to reply (depending on my travel schedule) but I try to make the best of the time I can put into my Ansible contributions.

I am not expecting from you, @gundalow, to fix this situation for me. I will rather write to the ansible-devel mailing list to ask for input on how I should structure my work to maintain the modules. I just think that this is an interesting perspective from a new contributor 🙂