EC2 Inventory Errors, but no error shown #13357

washburnjoe · 2022-12-20T16:12:55Z

Please confirm the following

I agree to follow this project's code of conduct.
I have checked the current issues for duplicates.
I understand that AWX is open source software provided for free and that I might not receive a timely response.

Bug Summary

Out of the box AWX install on Kubernetes. I created a credential for AWS with an Access Key and permissions to Read EC2. I created an inventory with source of Amazon EC2 and set the AWS credential. I have tried both warning, info and debug for the log type.

The job comes back with an error, but nothing of note in the logs and no hosts show up in inventory. Maybe I am not understanding how inventory works. As I understand it, I should get back every EC2 instance using the default settings.

 config file = None
  configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections
  executable location = /usr/local/bin/ansible-inventory
  python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)]
  jinja version = 2.11.3
  libyaml = True
No config file found; using defaults
redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2
Parsed /runner/inventory/aws_ec2.yml inventory source with auto plugin

AWX version

21.10.0

Select the relevant components

Installation method

kubernetes

Modifications

no

Ansible version

No response

Operating system

No response

Web browser

No response

Steps to reproduce

Create an IAM user with the built in EC2ReadOnly policy.
Create an IAM access key
Create a credential using that KEY and Secret
Make an Inventory and add a source of Amazon EC2 and set the credential you just made
sync

Expected results

I expect that I would get every EC2 instance loaded into inventory

Actual results

No inventory and the sync job has Error as the status

Log file

config file = None configured module search path = ['/home/runner/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.8/site-packages/ansible ansible collection location = /home/runner/.ansible/collections:/usr/share/ansible/collections:/usr/share/automation-controller/collections executable location = /usr/local/bin/ansible-inventory python version = 3.8.13 (default, Jun 24 2022, 15:27:57) [GCC 8.5.0 20210514 (Red Hat 8.5.0-13)] jinja version = 2.11.3 libyaml = True No config file found; using defaults redirecting (type: inventory) ansible.builtin.aws_ec2 to amazon.aws.aws_ec2 Parsed /runner/inventory/aws_ec2.yml inventory source with auto plugin

Additional information

No response

The text was updated successfully, but these errors were encountered:

washburnjoe · 2022-12-20T22:27:15Z

Logs from task pod

2022-12-20 22:23:27,932 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 acknowledged
2022-12-20 22:23:28,019 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 waiting
2022-12-20 22:23:28,223 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 pre run
2022-12-20 22:23:28,227 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 preparing playbook
2022-12-20 22:23:28,256 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 running playbook
2022-12-20 22:23:28,280 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 work unit id received
2022-12-20 22:23:28,291 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 work unit id assigned
2022-12-20 22:23:47,633 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 post run
2022-12-20 22:23:47,802 INFO [7d97167f01bb47108a3895fb0fee2c91] awx.analytics.job_lifecycle inventoryupdate-5 finalize run
2022-12-20 22:23:47,812 WARNING [7d97167f01bb47108a3895fb0fee2c91] awx.main.dispatch inventory_update 5 (error) encountered an error (rc=None), please see task stdout for details.

washburnjoe · 2022-12-21T02:19:49Z

I did a find on the EE and the task pod for the plugin and cant find it.

/runner/inventory/aws_ec2.yml

Are the plugins not installed be default?

filipprosovsky · 2022-12-21T10:21:48Z

Present in 21.10.2 as well.

filipprosovsky · 2022-12-21T15:50:52Z

result_traceback from api/v2/inventory_updates/<id>

res = receptor_job.run()\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 319, in run\n    
res = self._run_internal(receptor_ctl)\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 394, in _run_internal\n    
res = processor_future.result()\n  File \"/usr/lib64/python3.9/concurrent/futures/_base.py\", line 446, in result\n    
return self.__get_result()\n  File \"/usr/lib64/python3.9/concurrent/futures/_base.py\", line 391, in __get_result\n    
raise self._exception\n  File \"/usr/lib64/python3.9/concurrent/futures/thread.py\", line 58, in run\n    result = self.fn(*self.args, 
**self.kwargs)\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/common.py\", line 1214, in wrapper_cleanup_new_process\n    
return func(*args, **kwargs)\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 453, in processor\n    
return ansible_runner.interface.run(\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/interface.py\", line 210, in run\n   
 r.run()\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py\", line 254, in run\n    self.artifacts_callback(data)\n  
 File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py\", line 233, in artifacts_callback\n    
 unstream_dir(self._input, length, self.artifact_dir)\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/streaming.py\", line 71, in unstream_dir\n    
 data = source.read(chunk_size)\n  File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/base64io.py\", line 282, in read\n    
 results.write(base64.b64decode(data))\n  File \"/usr/lib64/python3.9/base64.py\", line 87, in b64decode\n    return binascii.a2b_base64(s)\nbinascii.Error: Incorrect padding\n",```

fosterseth · 2022-12-21T16:23:53Z

@filipprosovsky "return binascii.a2b_base64(s)\nbinascii.Error: Incorrect padding\n",```" are you seeing that message regularly? which k8s version are you on?

fosterseth · 2022-12-21T16:29:47Z

@washburnjoe does result_traceback for that failed inventory update show a similar error to what @filipprosovsky 's shows?

filipprosovsky · 2022-12-21T16:32:15Z

This will be possibly related to #11338 receptor issue, which is fixed in ansible/receptor#683 in K8S version 1.24.8
We are on EKS, k8s version 1.24.7, will try to higher the max log size in kubelet and test

washburnjoe · 2022-12-21T16:33:58Z

@fosterseth Unsure, I uninstalled last night and put in 21.9.0 because I thought the issue might have been part of #13313

I still had the problem with 21.9.0

And now I just uninstalled 21.9.0 to try another older version. I will report back ASAP

filipprosovsky · 2022-12-21T16:36:05Z

We are using awx-operator, and there is a problem with the charts as older are not available. But I downloaded previously working release 1.0.0 and uploaded it to our ArgoCD and deployed older AWX 21.9.0 version. Same thing happening. The thing is everything was working, but before upgrade to 1.24, so I assume this will be most likely related to cases I mentioned above. At least in our case.

washburnjoe · 2022-12-21T17:07:15Z

I rolled back to operator 0.30.0 and AWX 21.7.0. So I am in agreement it seems to be related more to Kubernetes. Because the same problem exists. We are using Kube 1.24

result_traceback

raceback (most recent call last):\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/jobs.py\", line 598, in run\n res = receptor_job.run()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 306, in run\n res = self._run_internal(receptor_ctl)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 381, in _run_internal\n res = processor_future.result()\n File \"/usr/lib64/python3.9/concurrent/futures/_base.py\", line 446, in result\n return self.__get_result()\n File \"/usr/lib64/python3.9/concurrent/futures/_base.py\", line 391, in __get_result\n raise self._exception\n File \"/usr/lib64/python3.9/concurrent/futures/thread.py\", line 58, in run\n result = self.fn(*self.args, **self.kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/utils/common.py\", line 1188, in wrapper_cleanup_new_process\n return func(*args, **kwargs)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/awx/main/tasks/receptor.py\", line 440, in processor\n return ansible_runner.interface.run(\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/interface.py\", line 210, in run\n r.run()\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py\", line 254, in run\n self.artifacts_callback(data)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/streaming.py\", line 233, in artifacts_callback\n unstream_dir(self._input, length, self.artifact_dir)\n File \"/var/lib/awx/venv/awx/lib64/python3.9/site-packages/ansible_runner/utils/streaming.py\", line 70, in unstream_dir\n with zipfile.ZipFile(tmp.name, \"r\") as archive:\n File \"/usr/lib64/python3.9/zipfile.py\", line 1266, in __init__\n self._RealGetContents()\n File \"/usr/lib64/python3.9/zipfile.py\", line 1333, in _RealGetContents\n raise BadZipFile(\"File is not a zip file\")\nzipfile.BadZipFile: File is not a zip file\n",

filipprosovsky · 2022-12-21T23:46:47Z

@fosterseth increasing the container max log size did not help. Still getting the same. I set RECEPTOR_RELEASE_WORK to false and when I check logs of automation pod, I do see some base64-like long string in there:

{"status": "successful", "runner_ident": "4522"}
{"zipfile": 16974}
UEsDBBQAAAAIABi9lVXH5bMNEwQAAJILAAAHAAAAY29tbWFuZK1WXXPaOhD9Kx2er21CSHH9JmyFuBjLIwtSptPRCMdNnBibsQ1t0un etc. 
{"eof": true}```

filipprosovsky · 2022-12-22T08:28:15Z

@filipprosovsky "return binascii.a2b_base64(s)\nbinascii.Error: Incorrect padding\n",```" are you seeing that message regularly? which k8s version are you on?

@fosterseth 1.24 and yea, we cannot sync the dynamic inventory in aws using aws_ec2 plugin, as we get this error everytime we run the sync. So it is not working at all for us.

washburnjoe · 2022-12-22T22:04:21Z

Maybe related to ansible/ansible-runner#998

filipprosovsky · 2023-01-09T07:48:08Z

ansible/receptor#683 This works. Can be closed

shanemcd · 2023-01-09T08:06:02Z

Thanks for letting us know @filipprosovsky .

github-actions bot added component:ui needs_triage type:bug community labels Dec 20, 2022

shanemcd closed this as completed Jan 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EC2 Inventory Errors, but no error shown #13357

EC2 Inventory Errors, but no error shown #13357

washburnjoe commented Dec 20, 2022

washburnjoe commented Dec 20, 2022 •

edited

washburnjoe commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022 •

edited

fosterseth commented Dec 21, 2022

fosterseth commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022 •

edited

washburnjoe commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

washburnjoe commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 22, 2022 •

edited

washburnjoe commented Dec 22, 2022

filipprosovsky commented Jan 9, 2023 •

edited

shanemcd commented Jan 9, 2023

EC2 Inventory Errors, but no error shown #13357

EC2 Inventory Errors, but no error shown #13357

Comments

washburnjoe commented Dec 20, 2022

Please confirm the following

Bug Summary

AWX version

Select the relevant components

Installation method

Modifications

Ansible version

Operating system

Web browser

Steps to reproduce

Expected results

Actual results

Additional information

washburnjoe commented Dec 20, 2022 • edited

washburnjoe commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022 • edited

fosterseth commented Dec 21, 2022

fosterseth commented Dec 21, 2022

filipprosovsky commented Dec 21, 2022 • edited

washburnjoe commented Dec 21, 2022 • edited

filipprosovsky commented Dec 21, 2022 • edited

washburnjoe commented Dec 21, 2022 • edited

filipprosovsky commented Dec 21, 2022 • edited

filipprosovsky commented Dec 22, 2022 • edited

washburnjoe commented Dec 22, 2022

filipprosovsky commented Jan 9, 2023 • edited

shanemcd commented Jan 9, 2023

washburnjoe commented Dec 20, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

washburnjoe commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

washburnjoe commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 21, 2022 •

edited

filipprosovsky commented Dec 22, 2022 •

edited

filipprosovsky commented Jan 9, 2023 •

edited