'./start_asterisk start' works at CLI—but ansible 2.15.0 freezes trying to wrap up running the same command—unless it's already been run manually! [Ugly Hack Works: run it as an async background task & wait 30 sec] [WAS: Break apart FreePBX install into 2 distinct Ansible steps] [WAS: Experimentally eliminate freepbx.yml blocking on presence of /var/www/html/freepbx] #3588

holta · 2023-05-21T15:13:19Z

Ongoing experiment with @EMG70 to try to eliminate freezing during install of FreePBX, that suddently appeared in recent days.

Possibly due to ansible-core 2.15.0 [??] or upstream change(s) in Asterisk or FreePBX [MAYBE ALSO?]

holta · 2023-05-21T15:46:12Z

This [original] PR [did not] help. Quite mysterious, as the equivalent manual commands seem to work fine:

sudo su -
cd /opt/iiab/freepbx
./start_asterisk start
./install -n --webroot /var/www/html/freepbx --dbuser asterisk --dbpass asterisk

Which very strongly suggests... this might be arising from the new ansible-core 2.15.0: [IN PART??]

PR Recommend ansible-core 2.15.0 #3577

holta · 2023-05-21T21:04:15Z

Not yet solved, so just a few details around how this is failing:

If ./start_asterisk start is run manually, it works every time (in about ~6 seconds) and Ansible can continue flawlessly, installing freePBX to completion.
However ansible-core 2.13.8 and 2.14.3 and 2.15.0 cannot get ./start_asterisk start to fully complete and return (bash breakpoints prove the bash script [ start_asterisk ] runs to completion, but Ansible somehow refuses to acknowledge this and continue!) Leading to freepbx.yml stalling, every time. No matter whether Asterisk is running a priori or not (just FYI in the normal workflow, Asterisk is NOT running at this point).
If Ansible is instructed to run ./start_asterisk start with async: 30 (to forcibly end the "stubborn job" after 30 seconds) this causes damage, preventing subsequent jobs from running properly. And once again, manually running ./start_asterisk start mysteriously fixes everything, allowing all Ansible steps in freepbx.yml to complete flawlessly.
start_asterisk invokes https://github.com/asterisk/asterisk/blob/master/contrib/scripts/safe_asterisk but that hasn't changed in 7 years (so it's unlikely to be the cause!) These 2 commands can be very useful, showing what's happening when/after ./start_asterisk start is run:
- pidof asterisk
- ps -ef | grep asterisk

…ue BUT...

…now)

holta · 2023-05-22T00:39:09Z

@EMG70 this PR is an ugly hack (in my opinion) but it does work.

Please test it carefully Monday if you can, as explained here:

PR Tips for very rapid installs & testing of IIAB + Asterisk/FreePBX (using Multipass VM's) #3589

Thanks!

holta · 2023-05-22T01:04:35Z

@EMG70 we still don't know what's causing this sudden regression (Ansible blocking/freezing completely) in recent days or weeks.

▶️ If you can confirm whether this bug is also affecting 64-bit Raspberry Pi OS (not just Debian 11 PC's/VM's) that would be extremely helpful! ◀️

As we really should be more precise than papering this over — identifying if this is due to an upstream change within Asterisk or FreePBX — or something more OS-related very hypothetically.

holta · 2023-05-22T02:47:42Z

Useful Data Point:

This bug definitely did not occur on 2023-05-07 (just 2 weeks ago) when @EMG70 kept careful notes of an IIAB and Asterisk/FreePBX install onto Debian 11.

Conclusion: Something serious changed... between May 7th and May 20th... even if we have no idea what just yet 🤔

(Next up, it would be great to ascertain one way of the other...)

whether this bug is also affecting 64-bit Raspberry Pi OS (not just Debian 11 PC's/VM's)

holta · 2023-05-22T04:54:34Z

@EMG70 confirmed:

64-bit RasPiOS (on RPi 400) is affected in the exact same way as Debian 11.
This PR ('./start_asterisk start' works at CLI—but ansible 2.15.0 freezes trying to wrap up running the same command—unless it's already been run manually! [Ugly Hack Works: run it as an async background task & wait 30 sec] [WAS: Break apart FreePBX install into 2 distinct Ansible steps] [WAS: Experimentally eliminate freepbx.yml blocking on presence of /var/www/html/freepbx] #3588) likewise works around the problem on 64-bit RasPiOS (on RPi 400), allowing FreePBX to successfully install.
iiab-diagnostics confirming both above: http://sprunge.us/wahbCm?en

Conclusions:

Rapid-testing using Debian 11 VM's is indeed representative (of other OS's and slower HW) here.
It appears likely that Asterisk or FreePBX pushed a nuisance change in one of their most recent point releases — in the last ~14 days — whether on purpose or not...

holta · 2023-05-22T05:00:18Z

Suggestion:

While this PR definitely mitigates the problem, let's ask around over the coming days, to see if the actual problem can be understood better.
Ideally with a less hacky solution and a better understanding of why running ./start_asterisk start from the command-line behaves so differently from invoking the exact same command via Ansible.

holta · 2023-05-22T13:05:34Z

This bug definitely did not occur on 2023-05-07 (just 2 weeks ago)

Counterintuitively:

Asterisk hasn't changed since 2023-04-03 when 20.2.1 was released:
- https://downloads.asterisk.org/pub/telephony/asterisk/
- https://downloads.asterisk.org/pub/telephony/asterisk/old-releases/asterisk-20.2.1.tar.gz
If one believes GitHub (maybe a bad idea) FreePBX hasn't changed since 2023-03-15 when 16.0.39 was released:
- https://github.com/FreePBX/framework/commits/release/16.0
- But 16.0.40.3 is mysteriously installed below, suggesting GitHub may now be a stale mirror of Sangoma's main (perhaps hidden) repo ?!

Confusing New Data Point — for the record:

I installed IIAB + PBX onto Debian 11 with ansible-core 2.14.3 (WITHOUT this PR './start_asterisk start' works at CLI—but ansible 2.15.0 freezes trying to wrap up running the same command—unless it's already been run manually! [Ugly Hack Works: run it as an async background task & wait 30 sec] [WAS: Break apart FreePBX install into 2 distinct Ansible steps] [WAS: Experimentally eliminate freepbx.yml blocking on presence of /var/www/html/freepbx] #3588) and it got (A BIT) further — Ansible mysteriously did not freeze with ./start_asterisk start (!) — but then failed 4 steps below that:

root@deb11:~# fwconsole ma downloadinstall callrecording cdr conferences core customappsreg dashboard featurecodeadmin infoservices logfiles music pm2 recordings sipsettings soundlang voicemail
No repos specified, using: [standard] from last GUI settings

Downloading module 'callrecording'
Processing callrecording
Verifying local module download...Verified
Extracting...Done
Download completed in 0 seconds
Updating tables callrecording, callrecording_module...Done
Generating CSS...Done
Module callrecording version 16.0.20 successfully installed
Downloading module 'cdr'
Processing cdr
Verifying local module download...Verified
Extracting...Done
Download completed in 0 seconds
Checking if field cnum is present in cdr table..OK!
Checking if field cnam is present in cdr table..OK!
Checking if field outbound_cnum is present in cdr table..OK!
Checking if field outbound_cnam is present in cdr table..OK!
Checking if field dst_cnam is present in cdr table..OK!
Checking if field linkedid is present in cdr table..OK!
Checking if field peeraccount is present in cdr table..OK!
Checking if field sequence is present in cdr table..OK!
Error: Call to undefined function writeCustomFiles() in file /var/www/html/freepbx/admin/modules/cdr/install.php on line 181
Stack trace:
  1. Error->() /var/www/html/freepbx/admin/modules/cdr/install.php:181
  2. include_once() /var/www/html/freepbx/admin/libraries/modulefunctions.class.php:2663
  3. module_functions->_doinclude() /var/www/html/freepbx/admin/libraries/modulefunctions.class.php:2579
  4. module_functions->_runscripts() /var/www/html/freepbx/admin/libraries/modulefunctions.class.php:2045
  5. module_functions->install() /var/www/html/freepbx/admin/libraries/Console/Moduleadmin.class.php:444
  6. FreePBX\Console\Command\Moduleadmin->doInstall() /var/www/html/freepbx/admin/libraries/Console/Moduleadmin.class.php:1578
  7. FreePBX\Console\Command\Moduleadmin->handleArgs() /var/www/html/freepbx/admin/libraries/Console/Moduleadmin.class.php:236
  8. FreePBX\Console\Command\Moduleadmin->execute() /var/www/html/freepbx/admin/libraries/Composer/vendor/symfony/console/Command/Command.php:255
  9. Symfony\Component\Console\Command\Command->run() /var/www/html/freepbx/admin/libraries/Composer/vendor/symfony/console/Application.php:992
 10. Symfony\Component\Console\Application->doRunCommand() /var/www/html/freepbx/admin/libraries/Composer/vendor/symfony/console/Application.php:255
 11. Symfony\Component\Console\Application->doRun() /var/www/html/freepbx/admin/libraries/Composer/vendor/symfony/console/Application.php:148
 12. Symfony\Component\Console\Application->run() /var/lib/asterisk/bin/fwconsole:163

Almost as if ansible-core 2.15.0 released on 2023-05-15 (might also?) be implicated after all ?
The Mystery Deepens, as trying to repeat the prior 4 steps (which during all other tests in recent days had been rock solid from the CLI) suddenly no longer works in this specific test 😒
- ./start_asterisk start # WORKS
- ./install -n --webroot /var/www/html/freepbx --dbuser asterisk --dbpass asterisk # WORKS, announcing "16.0.39 successfully installed"
- fwconsole ma downloadinstall framework # WORKS, announcing "16.0.40.3 successfully installed"
- But Then...

root@deb11:~# fwconsole reload
Reload Started

In Database.class.php line 247:

  SQLSTATE[42S02]: Base table or view not found: 1146 Table 'asterisk.voicemail_admin' doesn't exist


reload [--json] [--dry-run] [--skip-registry-checks] [--dont-reload-asterisk]

VERY PRELIM HUNCHES:

Is it possible ansible-core and upstream turmoil within FreePBX are both now implicated?
Yet more testing is needed, to tease out what's really causing the above (entirely new kind of) FreePBX progress AND failure.
If upstream turmoil within ansible-core and/or FreePBX appear strongly implicicated in the end, then waiting a month-or-so might also (possibly) be revealing.

holta · 2023-05-22T13:22:22Z

Here's my latest iiab-pbx.yml just in case it benefits others doing similar rapid-testing:

# https://cloudinit.readthedocs.io/en/latest/topics/examples.html

runcmd:
  - apt update
  - apt install git emacs -y
  - mkdir /etc/iiab
  - curl https://raw.githubusercontent.com/iiab/iiab/master/vars/local_vars_unittest.yml > /etc/iiab/local_vars.yml
  - [sed, -i, 's/^pbx_install:.*/pbx_install: True/', /etc/iiab/local_vars.yml]
  - [sed, -i, 's/^pbx_enabled:.*/pbx_enabled: True/', /etc/iiab/local_vars.yml]
  #- curl iiab.io/risky.txt | bash -s 3588 &
  - mkdir /opt/iiab
  - cd /opt/iiab
  - git clone https://github.com/iiab/iiab
  - cd iiab
  - git switch -c pbx-test
  #- scripts/ansible
  #- ./iiab-install

Debian 11 VM's can then be created nearly instantly as follows:

multipass launch https://cloud.debian.org/images/cloud/bullseye/latest/debian-11-generic-amd64.qcow2 -n deb11 -c 2 -m 2G -d 25G --bridged --cloud-init iiab-pbx.yml && multipass shell deb11

Just remember to never reboot a Debian 11 VM under Multipass !

holta · 2023-05-22T14:39:06Z

New data points... roles/pbx seems to install (beginning to end) on ansible-core 2.14.3 OR 2.14.5 — when these versions of Ansible are installed from the very beginning on Debian 11!

My earlier tests of ansible-core 2.13.8 and 2.14.3 regrettably appear flawed...
- LESSON LEARNED: Quite possibly Ansible Collections also need to be changed, whenever one swaps from one version of ansible-core to another!
So it would appear ansible-core 2.15.0 is the primary culprit... 😮

Preliminary Conclusions:

While this PR (3588) works to mitigate ansible-core 2.15.0's mess, I will report this upstream, so that ansible-core 2.15.1+ (hopefully!) fixes this.

holta · 2023-05-22T16:51:06Z

ASIDE 1: Let's hope ansible-core 2.5.1+ in "June" works far better!?

ASIDE 2: Quite separately, this original PR (3588) had value in narrowly eliminating freepbx.yml blocking on presence of /var/www/html/freepbx — this should move forward as a separate PR. As unlike in 2020/2021, FreePBX no longer fails when freepbx.yml is re-run on top of a partial (or complete?) FreePBX installation ✅

TEMPORARILY revert ansible-core 2.15.0 to 2.14.6 til the dust settles — e.g. for FreePBX #3588

holta · 2023-05-22T17:51:03Z

FYI Internet-in-a-Box has temporarily reverted from ansible-core 2.15.0 to 2.14.6 — allowing FreePBX installs to proceed unimpeded:

PR TEMPORARILY revert ansible-core 2.15.0 to 2.14.6 til the dust settles — e.g. for FreePBX #3588 #3590

Eliminate freepbx.yml blocking on presence /var/www/html/freepbx

d17f6ea

holta added the question label May 21, 2023

holta added this to the 8.1 milestone May 21, 2023

Break apart FreePBX install...into 2 Ansible steps

6386900

holta changed the title ~~Experimentally eliminate freepbx.yml blocking on presence of /var/www/html/freepbx~~ Break apart FreePBX install into 2 distinct Ansible steps [WAS: Experimentally eliminate freepbx.yml blocking on presence of /var/www/html/freepbx] May 21, 2023

holta added the bug label May 21, 2023

'./start_asterisk start' with 'async: 30' can force Ansible to contin…

6622fee

…ue BUT...

holta mentioned this pull request May 21, 2023

Tips for very rapid installs & testing of IIAB + Asterisk/FreePBX (using Multipass VM's) #3589

Open

'./start_asterisk start' async background task (ugly hack, works for …

ea95dd4

…now)

holta marked this pull request as ready for review May 22, 2023 00:37

pbx/tasks/freepbx.yml: Mention PR iiab#3588

8956b52

holta added the help wanted label May 22, 2023

holta mentioned this pull request May 22, 2023

command / shell never return as of 2.15.0 when stdout/stderr remain held open ansible/ansible#80863

Closed

1 task

holta added a commit to holta/iiab that referenced this pull request May 22, 2023

TEMPORARILY revert ansible-core 2.15.0 to 2.14.6 for iiab#3588

56d2733

holta mentioned this pull request May 22, 2023

TEMPORARILY revert ansible-core 2.15.0 to 2.14.6 til the dust settles — e.g. for FreePBX #3588 #3590

Merged

holta added a commit that referenced this pull request May 22, 2023

Merge pull request #3590 from holta/ansible-core-2.14.6

2973b28

TEMPORARILY revert ansible-core 2.15.0 to 2.14.6 til the dust settles — e.g. for FreePBX #3588

holta mentioned this pull request Jun 1, 2023

Test ansible-core 2.15.x hoping the serious regression in 2.15.0 & 2.15.1 is fixed soon in mid-2023 [WAS: Test ansible-core 2.15 Release Candidate(s) [ETA 2023-05-15]] #3552

Closed

holta mentioned this pull request Jun 20, 2023

Recommend ansible-core 2.15.1 [and interim workaround to start Asterisk, so FreePBX can install] #3604

Merged

holta mentioned this pull request Aug 14, 2023

scripts/ansible: Recommend ansible-core 2.15.3 #3620

Merged

holta closed this Nov 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

holta commented May 21, 2023 •

edited

holta commented May 21, 2023 •

edited

holta commented May 21, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

Conversation

holta commented May 21, 2023 • edited

holta commented May 21, 2023 • edited

holta commented May 21, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 22, 2023

holta commented May 22, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 22, 2023

holta commented May 22, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 22, 2023 • edited

holta commented May 21, 2023 •

edited

holta commented May 21, 2023 •

edited

holta commented May 21, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited

holta commented May 22, 2023 •

edited