I figured out what's happening here, but not how to fix it. There is a flaw in the smart fact gathering design.

At the start of a multi-play playbook run such as the typical site.yml, if previously cached facts have not expired, they will not be regathered. Then, if they expire later during the playbook run, no attempt is made to refresh them, and the undefined variable error occurs.

From the documentation "The value ‘smart’ means each new host that has no facts discovered will be scanned, but if the same host is addressed in multiple plays it will not be contacted again in the playbook run."

It wouldn't really matter what timeout value is used for fact_caching_timeout, the potential for this condition still exists.

Evidence that this happened. Running 2.0.1.0 now, turned fact caching back on to test, issue reappears.

One of the hosts that just failed, with an undefined ansible_system fact

12:33:54            TASK [splunk : install splunk] *************************************************
12:33:54            Monday 09 May 2016  12:33:54 -0400 (0:00:00.535)       0:33:37.157 ************ 
12:33:54            fatal: [wdv-servicessbx]: FAILED! => {"failed": true, "msg": "The conditional check 'ansible_system == 'Linux'' failed. The error was: error while evaluating conditional (ansible_system == 'Linux'): 'ansible_system' is undefined\n\nThe error appears to have been in '/home/icansible/ansible/roles/splunk/tasks/main.yml': line 2, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n---\n- include: linux.yml\n  ^ here\n"}

Looking in the cached file directory, the failed server hadn't been updated in the most recent execution, which launched at 12:00, because the file was not quite old enough yet, it was 2 hours 46 minutes old, our timeout is set to 3 hours

-rw-rw----.  1 icansible icansible   926 May  9 09:14 wdv-servicessbx

But, the site run failed at 12:33 because the fact file was now over 3 hours (10,800 seconds) old.

The only behavior I can't explain, is why we never saw this problem in 1.9.4 but see it in frequently starting with 2.0

Changed issue title to more accurately reflect the issue, now that it's been identified.

Suspected culprit 0aa0183 which fixed #12722

Note: We are using file (json) caching. For anyone having this problem, until the commit makes the 2.1 release, a suggested workaround is to switch from smart to explicit, and explicitly call setup once at the start of your playbook.

I'm not entirely clear how redis or other caching systems are working, but I believe they will see the same issue, and the fix committed above only covers json caching behavior.

Generating my /etc/hosts file in ansible2 fails. But only when playbook runs for a few minutes and then generates the /etc/hosts file. When running task direct with corresponding tag the /etc/hosts is generated without any issue.

fatal: [host]: FAILED! => {"changed": false, "failed": true, "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'ansible_ssh_host'"}

ansible (2.1.0.0)
gathering = smart
fact_caching = redis
fact_caching_timeout = 86400