Issues with clank install

[lokI8]

Hello all,

I'm installing atmosphere using clank in a 16.04 Ubuntu vm, and we've hit a few issues that we thought you guys might want to know about.

The atmosphere repo cloned to /opt/dev/atmosphere has uWSGI version 2.0.9 in it's requirements.txt. We are hitting a compilation error on this package described here: unbit/uwsgi#1262. We've tried changing the version number to 2.0.13, but then clank won't let us pass the cloning of the atmosphere repo, as we've made modifications and it can't pull. We're trying on an older version of Ubuntu now.

Two more issues you guys might want to know about:

1. We had to hard code the version number of postgresql to 9.5 in the main.yml of setup-postgres/tasks, as changing the variable in any of the files in var did not propagate to the ansible role, and the version of postgres automatically installed from clank was 9.5.
2. We had to manually install ruby 1.9.1 and comment out the line in the requirements file. There is no package matching that exact name on this version of Ubuntu, and apt-get does a reg-ex match to install the correct packages, so when the installer was unable to find it, it error'd out.

[xuhang57]

Adding to that, Jon and I tried to switch from Ubuntu 16.04 to 14.04 LTS and we passed those above errors. During the installation, we add a global variable in dist_files\variables.yml to fix an error that `TASK [app-generate-ini-config : create local variables.ini.ini from dict] ******
fatal: [localhost -> localhost]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'MOCK_USERNAME' is undefined"}``

However after that, we failed at the stage **: TASK [app-generate-ini-config : run generate script for app] *********************

The specific errors we are getting are
Undeclared variables found in /opt/dev/atmosphere/extras/init.d/celeryd.default.dist.j2: \n\t- FLOWER_EMAIL_ACCESS_LIST\n\t- FLOWER_OAUTH2_REDIRECT_URI\n\t- FLOWER_OAUTH2_CLIENT_ID\n\t- FLOWER_AUTH_SCHEME\n\t- FLOWER_BASIC_PASSWORD\n\t- FLOWER_OAUTH2_CLIENT_SECRET\n\t- FLOWER_BASIC_USERNAME

And some other errors like Unused variables found in section local.py on auth parameters even though we set MOC_AUTH as True in the variables.yml file.

Apologize for throwing so many installation errors at one time. It would be much appreciated if any of you can point us the direction of fix them. Thanks for your help!

[lenards]

@xuhang57 @lokI8 - I can get you basic values for the FLOWER_* variables - that lacking from dist_files/variables.yml is an oversight.

[lenards]

@xuhang57 no apologies necessary - thank you for providing details for us.

[lenards]

@lokI8 & @xuhang57

ATMO:
    # .... 
    celeryd.default:
        # ... 
        FLOWER_AUTH_SCHEME: "basic"
        # Required if auth_scheme is 'basic'
        FLOWER_BASIC_USERNAME: "<username-redacted>"
        FLOWER_BASIC_PASSWORD: "<password-redacted>"
        # Required if auth_scheme is 'google' 
        FLOWER_EMAIL_ACCESS_LIST:
        FLOWER_OAUTH2_CLIENT_ID:
        FLOWER_OAUTH2_CLIENT_SECRET:
        FLOWER_OAUTH2_REDIRECT_URI:

[lenards]

To Lucas' feedback:

• MOCK_USERNAME missing from variables.yml
• FLOWER_* missing from variables.yml

commit f14a9a7 resolves this as much as it communicates the need for the variables in the distribution file.

[lenards]

To Jonathan's (@lokl8) feeback:

As you noted, we are using 14.04.3 so we haven't hit this issue with uWSGI yet. Thank you for sharing this, as it'll be of great assistance as we shift to 16.04.

For context/reference:

The atmosphere repo cloned to /opt/dev/atmosphere has uWSGI version 2.0.9 in it's requirements.txt. We are hitting a compilation error on this package described here: unbit/uwsgi#1262. We've tried changing the version number to 2.0.13, but then clank won't let us pass the cloning of the atmosphere repo, as we've made modifications and it can't pull. We're trying on an older version of Ubuntu now.

I will open an issue in the Atmosphere Repository on GitHub so we can investigate updating to 2.0.13 as a resolution.

@lokI8 - within the variables.yml provided to Clank, what are is used for ATMOSPHERE_BRANCH?

That way I can target a potential fix to that branch.

1. We had to hard code the version number of postgresql to 9.5 in the main.yml of setup-postgres/tasks, as changing the variable in any of the files in var did not propagate to the ansible role, and the version of postgres automatically installed from clank was 9.5.

Okay - this is great feedback. Our intent is to provide DB_VERSION as a sensible default within role/setup-postgres. Could you confirm the version of Ansible you are using?

2. We had to manually install ruby 1.9.1 and comment out the line in the requirements file. There is no package matching that exact name on this version of Ubuntu, and apt-get does a reg-ex match to install the correct packages, so when the installer was unable to find it, it error'd out.

Did you get this apt-get error in 16.04?

Some background, the ruby dependency is a base (or transitive) dependency for Redis, I believe.

[lokI8]

Hi Andrew (@lenards), thanks for the quick response! Greatly appreciated.

Our ATMOSPHERE_BRANCH is just set to "" in our variables.yml, I believe this means we're just working off master.

After looking through the variables.yml file again, it appears we missed the "DB_VERSION: 9.3" variable. I'd imagine this is why changing the defaults.yml and Ubuntu.yml in setup-postgres/vars didn't impact anything. Oops! We are using Ansible version 2.1.0.0.

Yes, that apt-get error for ruby was found in 16.04. There is no package for 16.04 ubuntu that is just flat "Ruby$version". Instead, it appears apt-get installs a set of packages that it reg-ex matches to "Ruby1.9.1". We didn't experience this error once we switched over to 14.04.

Our latest error, once adding the variables you provided, is: "Testing for preconditions...", "", "Local variables.ini missing section: celeryd.default", "", "", "Configuration files were not generated.", "", "FAILED. "]. This is bizarre, because we clearly have the celeryd.default section in our variables.yml file, under the ATMO header... we just added the few variables to the end of that section, as you specified.

[lenards]

@lokI8 - you are correct, ATMOSPHERE_BRANCH: "" will default to master.

[lenards]

@lokI8 - that error is unexpected regarding celeryd.default ... hmm, let me ponder some possible ways that might happen

[lenards]

For the error - you'll want to look at variables.ini (it might be variables.ini.ini, which is not intended name) under $CLANK_HOME/roles/app-generate-ini-config/files to see if that file is missing celeryd.default under ATMO.

Also, I gave you a slight misleading outline for those variables. But having more than one celeryd.default might generate an error as well.

[lenards]

For Clank - we have only verified and tested it with Ansible 2.0. I have experienced problems with Ansible 2.1 and passing along variables to parameterized roles (see Ansible Issues 16320). So that may (unconfirmed) lead to the issue with DB_VERSION - it's on me, of course, to confirm (have not yet).

[amercer1]

Hello @lokI8! Thank you for being patient as we flesh out these issues. Like @lenards said above, we have yet to test this on 16.04. In regards to the the DB_VERSION, it defaults to 9.3. If you wish to change to 9.5 without having to change the role vars, you can modify this variable in the variables.yml. This should propagate downwards into the role. Please let us know if this does not work. I will add an Ubuntu 16.04 var file to the setup-postgres role to automatically do this for us.

[lokI8]

The file is indeed named variables.ini.ini. Should I just mv the file, or should I fix a typo somewhere else?

Also, there is no ATMO header before all of the sections, but there is a "cleryd.default" section. Should I change this to "celeryd.default" as well? Is this file generated, or static?

[amercer1]

@lokI8 I added a quick fix to the setup-postgres role to account for db version difference. This fix should prevent you having to modify that role by hand going forward. Thanks again.

[lenards]

@lokI8 The ATMO header will not be present within variables.ini(.ini). That's only a convention for variables.yml to handle communicating the component the sections are associated with (in short, we don't expect it there).

"cleryd.default" is definitely a typo. That is likely the cause of the "missing section" error. I'd check the variables.yml in use to see if it has the typo as well.

[lenards]

I just want to assess where this is: @lokI8 @xuhang57

I believe the remaining issue is to address:

Yes, that apt-get error for ruby was found in 16.04. There is no package for 16.04 ubuntu that is just flat "Ruby$version". Instead, it appears apt-get installs a set of packages that it reg-ex matches to "Ruby1.9.1". We didn't experience this error once we switched over to 14.04.

Which I am guessing that 16.04 doesn't have a 1.9 dependency because Ruby 2.0+ is what projects likely should be using. Perhaps our defined Redis PPA is contributing to that. We will investigate.

Was the following issue with variables.ini and the ./configure script for Atmosphere repo resolved?

Our latest error, once adding the variables you provided, is: "Testing for preconditions...", "", "Local variables.ini missing section: celeryd.default", "", "", "Configuration files were not generated.", "", "FAILED. "]. This is bizarre, because we clearly have the celeryd.default section in our variables.yml file, under the ATMO header... we just added the few variables to the end of that section, as you specified.

[lenards]

@lokI8 Regarding:

The file is indeed named variables.ini.ini. Should I just mv the file, or should I fix a typo somewhere else?

The variables.ini.ini can be left as is - the Ansible role will copy it (with the extra .ini) into the correct location in ATMOSPHERE_LOCATION (likely set to /opt/dev/atmosphere by default).

What I would encourage you to do is verify the spelling of celeryd.default within your variables.yml file. I think that is the source of the typo because roles/app-generate-ini-config is iterating over the sections of ATMO to produce the variables.ini(.ini) (sans ATMO root heading).

[lokI8]

We did indeed have the same misspelling in our variables.yml as well. We've fixed that and have continued on.

The most recent error is during the app-django-manage-collectstatic role. We are getting a syntax error on line 192 of /opt/dev/atmosphere/atmosphere/settings/local.py, on the variable LOGSTASH_PORT =. I believe we need a value afterwords, if just the empty string, so it does not error. However, changing the code in /opt/dev/atmosphere makes ratchet unhappy at the git cloning stage. Should we create our own branch and fix this?

Thanks for all the help =)

[lenards]

@lokI8 - I was hoping not to give you the "you could clone Atmosphere & set the ATMOSPHERE_LOCATION to that" workaround. Before doing that, let me investigate a bit. I think if you are not using a logstash instance, you might be able to set it to LOGSTASH_PORT: 0 just to give it a valid integer to make Python happy.

[lenards]

Okay Jonathan - again, sorry about this. We have logstash configured in our environments, so the template doesn't handle this gracefully.

If you just give LOGSTASH_PORT: 0 then the local settings file generate will be valid Python:

# LOGSTASH
LOGSTASH_HOST = 'redact.make-redaction-cool-again.org'
LOGSTASH_PORT = 0
try:
    import logstash
    import threepio
    has_logstash = True
except ImportError:
    has_logstash = False

[xuhang57]

@lenards Thanks for your help. Adding the 0 to the LOGSTASH_PORT fixed the error and we continued on.

We now stuck on filling up the parameters in secrets.py. The most current error is also a syntax error indicating that IRODs is not defined and we notice that it needs up to provide OpenStack credential afterwards in secrets.py. Any dummy values you would recommend us to try to assign and it might fix the error?

Thanks so much!

[lenards]

Hi @xuhang57,

Empty variables for IRODS_* will allow you to move forward. I think the "null" YAML values will create an invalid Python syntax situation like you've encountered.

So - valid, empty/default values like this:

        IRODS_HOST: ''
        IRODS_PORT: 0
        IRODS_ZONE: ''
        IRODS_ADMIN_USER: ''
        IRODS_ADMIN_PASS: ''

Note: I believe that the secrets.py.j2 encloses these in single quotes, so in regression question previous - I believed "null" YAML values did produce valid Python. I'll get that re-evaluated

We have provided some context to the variables in [secrets.py] within the distribution file in the Atmosphere repository, perhaps that helps with the OpenStack credentials necessary. This aspect is an area we'll be documenting further.

Thank you for your persistence & patience.

[xuhang57]

@lokI8 @lenards We are having a debug session with Andy, Andre and Steve tomorrow. So I think we could close this issue now.

Thanks for all the help and we really appreciate it

[lenards]

Thank you Lance. I will close this. We appreciate all the feedback and patience.