Task "Generate cluster template file" fail

[okleber]

The task "Generate cluster template file" fail

#####Command line########
ansible-playbook /opt/cloudera-deploy/main.yml -i /opt/cloudera-deploy/cdppredev/inventory_static.ini -e "profile=/opt/cloudera-deploy/cdppredev/profile.yml" -e "definition_path=/opt/cloudera-deploy/cdppredev" -e "definition_file=definition.yml" -e "cluster_file=cluster.yml" -t full_cluster

#########Console error#########
TASK [cloudera.cluster.cluster : Generate cluster template file] ****************************************************************************************
task path: /opt/cldr-runner/collections/ansible_collections/cloudera/cluster/roles/deployment/cluster/tasks/create_base.yml:27
Thursday 27 May 2021 13:26:31 +0000 (0:00:00.060) 0:00:50.089 **********
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /root/.ansible/tmp"&& mkdir "echo /root/.ansible/tmp/ansible-tmp-1622121991.4248302-17965-148244685479104" && echo ansible-tmp-1622121991.4248302-17965-148244685479104="echo /root/.ansible/tmp/ansible-tmp-1622121991.4248302-17965-148244685479104" ) && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1622121991.4248302-17965-148244685479104/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
"changed": false,
"msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'bdengd704.cdadev.mydomain.com'"
}

#####inventory_static.ini####
[cloudera_manager]
bdengd002.cdadev.mydomain.com

[cluster_worker_nodes]
bdengd704.cdadev.mydomain.com

[cluster_worker_nodes:vars]
host_template=Workers

[cluster_master_nodes]
bdengd002.cdadev.mydomain.com host_template=Master1

[cluster:children]
cluster_master_nodes
cluster_worker_nodes

[db_server]
bdengd002.cdadev.mydomain.com

[deployment:children]
cluster
db_server

[deployment:vars]
ansible_user=root
ansible_ssh_private_key_file=~/.ssh/cldr_ssh_rsa

#########cluster.yml########

cloudera_manager_version: 7.1.5.0

clusters:

• name: CDPPREDEV
  services: [HDFS, YARN, ZOOKEEPER]
  repositories:
  • https://bdadmp01.zit.mydomain.com/cdp/cloudera-repos/cdh7/7.1.5.0/parcels/
    configs:
    HDFS:
    DATANODE:
    dfs_data_dir_list: /dfs/dn
    NAMENODE:
    dfs_name_dir_list: /dfs/nn
    SECONDARYNAMENODE:
    fs_checkpoint_dir_list: /dfs/snn
    YARN:
    RESOURCEMANAGER:
    yarn_scheduler_maximum_allocation_mb: 4096
    yarn_scheduler_maximum_allocation_vcores: 4
    NODEMANAGER:
    yarn_nodemanager_resource_memory_mb: 4096
    yarn_nodemanager_resource_cpu_vcores: 4
    yarn_nodemanager_local_dirs: /tmp/nm
    yarn_nodemanager_log_dirs: /var/log/nm
    GATEWAY:
    mapred_submit_replication: 3
    mapred_reduce_tasks: 6
    ZOOKEEPER:
    SERVICEWIDE:
    zookeeper_datadir_autocreate: true
    host_templates:
    Master1:
    HDFS: [NAMENODE, SECONDARYNAMENODE, HTTPFS]
    YARN: [RESOURCEMANAGER, JOBHISTORY]
    ZOOKEEPER: [SERVER]
    Workers:
    HDFS: [DATANODE]
    YARN: [NODEMANAGER]

mgmt:
name: Cloudera Management Service
services: [ALERTPUBLISHER, EVENTSERVER, HOSTMONITOR, REPORTSMANAGER, SERVICEMONITOR]

hosts:
configs:
host_default_proc_memswap_thresholds:
warning: never
critical: never
host_memswap_thresholds:
warning: never
critical: never
host_config_suppression_agent_system_user_group_validator: true

####profile.yml######
admin_password: "admin"
infra_type: "onpremise"

####definition.yml#####
datahub:
definitions:
- include: "datahub_streams_messaging_light.j2"

use_default_cluster_definition: no
use_download_mirror: no
preload_cm_parcel_repo: yes

###main.yml#####
I'm using cloudera-deploy repo

####cloudera runner docker v1.0.2####
I'm using docker container generated by quickstart.sh from cloudera-deploy repo

####My personal observations#######

If in inventory_static.ini I use the same host for Worker and Master, this task pass and the template start to be deployed to cloudera manager

[sdairs]

When you observe this failure, is your Cloudera Manager running? Can you go in to Cloudera Managers -> Hosts -> All Hosts - is the host bdengd704.cdadev.mydomain.com reported as a host in the list & heart beating? The most likely cause of this error is that the CM Agent install for the Worker node failed in a previous step, or it isn't heartbeating.

[okleber]

Hi, It happened when I tried to rerun the playbooks few times. I executed few times because I was still testing different parameters and getting errors. After complete cleaning(removal cloudera packages, directories used by hadoop and cloudera) from machines, the installation progress and I cant reproduce the issue anymore. Now the installation finish with success.

…

On Thu, 10 Jun 2021 at 11:59, Alasdair Brown ***@***.***> wrote: When you observe this failure, is your Cloudera Manager running? Can you go in to Cloudera Managers -> Hosts -> All Hosts - is the host bdengd704.cdadev.mydomain.com reported as a host in the list & heart beating? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#12 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACJVQWWJ2RLJSYIDJVYYAMDTSCEJBANCNFSM45UKH6CQ> .

[sdairs]

Cool thanks for confirming! We are going to improve checks & failures around this so its a bit cleaner

[AlexHff]

Hello there,
I am having the exact same error as described above. I have an inventory_static.ini file that looks like this:

[cloudera_manager]
master01.nikita.local

[cluster_master_nodes]
master01.nikita.local host_template=Master1

[cluster_worker_nodes]
worker01.nikita.local

[cluster_worker_nodes:vars]
host_template=Workers

[cluster:children]
cluster_master_nodes
cluster_worker_nodes

[db_server]
master01.nikita.local

[deployment:children]
cluster
db_server

[deployment:vars]
# Ansible will defer to the running SSH Agent for relevant keys
# Set the following to hardcode the SSH private key for the instances
# ansible_ssh_private_key_file=~/.ssh/mykey.pem  
ansible_user=vagrant

In my case, worker01.nikita.local is not reported as an existing host in the cluster but I can add it easily. In other words, it is reachable and I can connect from master01 to worker01 via ssh. As mentioned in @sdairs's response, it is likely that the CM Agent install for the node worker01 failed in a previous step.

Here is the command I am running:

ansible-playbook /opt/cloudera-deploy/main.yml -e "definition_path=examples/sandbox" -e "profile=/
opt/cloudera-deploy/profile.yml" -i /opt/cloudera-deploy/inventory_static.ini -t default_cluster

I tried destroying my nodes and starting them from the ground up, but this did not fix the issue. profile.yml is default except for the admin password. Is there any hint on how to go forward from there on? I can provide any additional information.

Update: I mentioned that I can connect via ssh from master01 to worker01 but when I try to add the host manually in CDM, the ssh connection fails. There is no real error log associated, therefore I cannot determine why it fails.

[sdairs]

@AlexHff You can manually install the CM agent & daemons package on the worker host, then edit the /etc/cloudera-scm-agent/config.ini file and point it at the CM server - then restart cloudera-scm-agent process. The host will eventually start heartbeating in CM then you can resume the playbook.

Did you notice any failures in your playbook run?
Where are you running the playbook from? Is it running on the CM node?

[AlexHff]

I noticed that the /etc/hosts files on my vms were poorly configured. More specifically, the FQDN mapped to 127.0.0.1. I fixed this and the deployed works perfectly now.

I wonder why I could connect from one host to another but CDM didn't accept this config.

[okleber]

as is not possible to reproduce, I'm closing the issue