[BUG] utils/jinja file client cache issue when instantiating two salt callers with different 'cachedir's

[jmarais]

Description
I instantiate two salt caller clients in the same python environment and it looks like the cached utils.jinja file_client is reused between them when resolving jinja imports.
This cause the second minion to fail when trying to resolve jinja imports. It seems like the file_client of the first salt minion is used and it validates that the imported file already exists in the first minion cachedir (and does not download the file to the second minion cache). The second minion will then produce an error because the file is not downloaded to the second minion cachedir. This also fail on the master branch.

I am trying to run salt states with test=true mode in order to validate salt/jinja file rendering before deploying. Each client is instantiated with all the 'cache' related config setup to point to a per minion directory.

Issue traced down to the reuse here:
https://github.com/saltstack/salt/blob/master/salt/utils/jinja.py#L106-L116

Does adding:

or self._file_client.opts["cachedir"] != self.opts["cachedir"]

to the if check there address the issue correctly?

Setup
Here is a repo with the example bug:
https://github.com/jmarais/salt-utils-jinja-cache-issue

git clone git@github.com:jmarais/salt-utils-jinja-cache-issue.git
cd salt-utils-jinja-cache-issue
make run

It comes down to trying to file.manage a first jinja fail with an import:

/path/to/destination.txt:
  file.managed:
    - source: salt://jinja_main.j2
    - template: jinja

jinja_main:

{%- from 'jinja_imported.j2' import foo -%}
what is the value: {{ foo.bar }}

jinja_imported:

{% set foo = namespace() %}
{% set foo.bar = "imported value" %}

Steps to Reproduce the behavior

git clone git@github.com:jmarais/salt-utils-jinja-cache-issue.git
cd salt-utils-jinja-cache-issue
make run

result:

end of log output

2022-08-22 10:31:51,732 salt.utils.templates DEBUG:Jinja Error
2022-08-22 10:31:51,732 salt.utils.templates DEBUG:Exception:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2
2022-08-22 10:31:51,733 salt.utils.templates DEBUG:Out:
2022-08-22 10:31:51,733 salt.utils.templates DEBUG:Line: 1
2022-08-22 10:31:51,733 salt.utils.templates DEBUG:TmplStr: {%- from 'jinja_imported.j2' import foo -%}
what is the value: {{ foo.bar }}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

2022-08-22 10:31:51,733 salt.utils.templates DEBUG:TraceStr: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

2022-08-22 10:31:51,733 salt.utils.templates ERROR:Rendering exception occurred
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 548, in render_jinja_tmpl
    "Jinja error: {}{}".format(exc, out), line, tmplstr, trace=tracestr
salt.exceptions.SaltRenderError: Jinja error: jinja_imported.j2
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

; line 1

---
{%- from 'jinja_imported.j2' import foo -%}    <======================
what is the value: {{ foo.bar }}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
[...]
---
2022-08-22 10:31:51,733 salt.state DEBUG:An exception occurred in this state: Jinja error: jinja_imported.j2
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

; line 1

---
{%- from 'jinja_imported.j2' import foo -%}    <======================
what is the value: {{ foo.bar }}

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
[...]
---
2022-08-22 10:31:51,744 salt.state ERROR:An exception occurred in this state: Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/state.py", line 2180, in call
    *cdata["args"], **cdata["kwargs"]
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1249, in wrapper
    return f(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/states/file.py", line 3026, in managed
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 149, in __call__
    return self.loader.run(run_func, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1201, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/contextvars/__init__.py", line 38, in run
    return callable(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 1216, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/usr/lib/python3.6/site-packages/salt/modules/file.py", line 5409, in check_managed_changes
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/modules/file.py", line 4661, in get_managed
    **kwargs
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 261, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 548, in render_jinja_tmpl
    "Jinja error: {}{}".format(exc, out), line, tmplstr, trace=tracestr
salt.exceptions.SaltRenderError: Jinja error: jinja_imported.j2
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/utils/templates.py", line 502, in render_jinja_tmpl
    output = template.render(**decoded_context)
  File "/usr/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 76, in render
    return original_render(self, *args, **kwargs)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 1008, in render
    return self.environment.handle_exception(exc_info, True)
  File "/usr/lib/python3.6/site-packages/jinja2/environment.py", line 780, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3.6/site-packages/jinja2/_compat.py", line 37, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/usr/lib/python3.6/site-packages/salt/utils/jinja.py", line 237, in get_source
    raise TemplateNotFound(template)
jinja2.exceptions.TemplateNotFound: jinja_imported.j2

This result can also be observed when looking at the resulting files in each minion cache directory:

find minion_cache result

$ find minion_cache/
minion_cache/
minion_cache/crab
minion_cache/crab/extmods
minion_cache/crab/cache
minion_cache/crab/cache/file_lists
minion_cache/crab/cache/file_lists/roots
minion_cache/crab/cache/file_lists/roots/base.p
minion_cache/crab/cache/roots
minion_cache/crab/cache/roots/mtime_map
minion_cache/crab/cache/roots/hash
minion_cache/crab/cache/roots/hash/base
minion_cache/crab/cache/roots/hash/base/jinja_imported.j2.hash.sha256
minion_cache/crab/cache/roots/hash/base/top.sls.hash.sha256
minion_cache/crab/cache/roots/hash/base/manage_jinja.sls.hash.sha256
minion_cache/crab/cache/roots/hash/base/jinja_main.j2.hash.sha256
minion_cache/crab/cache/files
minion_cache/crab/cache/files/base
minion_cache/crab/cache/files/base/top.sls
minion_cache/crab/cache/files/base/manage_jinja.sls
minion_cache/crab/cache/files/base/jinja_main.j2
minion_cache/crab/cache/files/base/jinja_imported.j2
minion_cache/crab/cache/highstate.cache.p
minion_cache/crab/cache/sls.p
minion_cache/crab/cache/module_refresh
minion_cache/crab/cache/accumulator
minion_cache/rock
minion_cache/rock/extmods
minion_cache/rock/cache
minion_cache/rock/cache/file_lists
minion_cache/rock/cache/file_lists/roots
minion_cache/rock/cache/file_lists/roots/base.p
minion_cache/rock/cache/roots
minion_cache/rock/cache/roots/mtime_map
minion_cache/rock/cache/roots/hash
minion_cache/rock/cache/roots/hash/base
minion_cache/rock/cache/roots/hash/base/jinja_imported.j2.hash.sha256
minion_cache/rock/cache/roots/hash/base/top.sls.hash.sha256
minion_cache/rock/cache/roots/hash/base/manage_jinja.sls.hash.sha256
minion_cache/rock/cache/roots/hash/base/jinja_main.j2.hash.sha256
minion_cache/rock/cache/files
minion_cache/rock/cache/files/base
minion_cache/rock/cache/files/base/top.sls
minion_cache/rock/cache/files/base/manage_jinja.sls
minion_cache/rock/cache/files/base/jinja_main.j2
minion_cache/rock/cache/highstate.cache.p
minion_cache/rock/cache/sls.p
minion_cache/rock/cache/module_refresh
minion_cache/rock/cache/accumulator

note the missing minion_cache/rock/cache/files/base/jinja_imported.j2 file for the 'rock' minion.

Expected behavior
I expect that each minion will be able to download template files to their own cache.

Versions Report

salt --versions-report

Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: Not Installed
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.4
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.15.0
        pygit2: Not Installed
        Python: 3.8.12 (default, Sep 21 2021, 00:10:52)
  python-gnupg: Not Installed
        PyYAML: 6.0
         PyZMQ: 21.0.2
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.3

System Versions:
          dist: centos 8
        locale: utf-8
       machine: x86_64
       release: 4.18.0-373.el8.x86_64
        system: Linux
       version: CentOS Stream 8

salt library __version__ number

$ python3 -c "import salt.version; print(salt.version.__version__)"
3004.2

Additional context
I am not sure if this is a bug or intended behaviour?

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!