Template provider via HTTP URL fails if given template is .json

[wieshka]

Subject of the issue

If you specify template as type http and uri with URL to JSON file, it is treated as YAML rather JSON file.

Your environment

• version of sceptre 4.0.2
• version of python Python 3.10.10
• which OS/distro MacOS

Steps to reproduce

Point template with type http to valid URL pointing at .JSON file
sceptre generate will yield output like this:

---
Contents of file

sceptre launch/create will fail with:

Traceback (most recent call last):
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/bin/sceptre", line 8, in <module>
    sys.exit(cli())
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/viestursproskins/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/cli/helpers.py", line 43, in decorated
    return func(*args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/cli/launch.py", line 62, in launch_command
    exit_code = launcher.launch(prune)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/cli/launch.py", line 105, in launch
    code = code or self._deploy(deploy_plan)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/cli/launch.py", line 197, in _deploy
    result = deploy_plan.launch()
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/plan.py", line 170, in launch
    return self._execute(*args)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/plan.py", line 29, in wrapped
    return func(self, *args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/plan.py", line 55, in _execute
    return executor.execute(*args)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/executor.py", line 52, in execute
    stack, status = future.result()
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 451, in result
    return self.__get_result()
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/opt/homebrew/Cellar/python@3.10/3.10.10/Frameworks/Python.framework/Versions/3.10/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/executor.py", line 59, in _execute
    result = getattr(actions, self.command)(*args)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
    response = func(self, *args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/actions.py", line 218, in launch
    status = self.update()
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/hooks/__init__.py", line 107, in decorated
    response = func(self, *args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/plan/actions.py", line 143, in update
    response = self.connection_manager.call(
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/connection_manager.py", line 54, in decorated
    return func(*args, **kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/sceptre/connection_manager.py", line 467, in call
    return getattr(client, command)(**kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/botocore/client.py", line 530, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/Users/<>/Library/Caches/pypoetry/virtualenvs/new-gen-3nxPWgEd-py3.10/lib/python3.10/site-packages/botocore/client.py", line 960, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the UpdateStack operation: Template format error: YAML not well-formed. (line 233, column 1)

Expected behaviour

Sceptre should correctly detect that given file is JSON

Actual behaviour

Sceptre fails to detect, processes as YAML, fails.

Due to the nature of the given .json template, I won't be able to share it this time.

[wieshka]

This better illustrates generate behavior:

[wieshka]

I can't reproduce with other JSON files, yet, given JSON which yields this error, works like a charm with Cloudformation directly. Will investigate more on my own, as I can't share the given JSON for security reasons.

[dboitnot]

Out of curiosity, do you get the same error if you remove the top --- line from the offending JSON file?

[sometxdude]

I am having the exact same problem trying to use a JSON template from the 's3' handler. It doesn't recognize that it is json and not yaml even though the extension is '.json'. NOTE: I do not have the '---' line in my json file.

[zaro0508]

@sometxdude @wieshka could you please try to reproduce this problem with a minimal json file and paste it here? thanks.

[zaro0508]

no response from @sometxdude and @wieshka so closing this issue

[wieshka]

Out of curiosity, do you get the same error if you remove the top --- line from the offending JSON file?

Given --- in output is only when using sceptre generate, in file it self there is no such string, and it is fully valid JSON file according to few tools I have checked with for any "smells".

Additionally, given JSON file works like a charm when used as template.path aka local file, and only when fetched from template.url (type http) it fails with error on latest Sceptre v4.3.0 on Python 3.11.6:

[2023-11-02 19:47:41] - landing-zone/external/soc - Creating Stack
"An error occurred (ValidationError) when calling the CreateStack operation: Template format error: YAML not well-formed. (line 259, column 1)".

@zaro0508 please re-open the case, I will continue to investigate the case.

[zaro0508]

The http handler code indicates that the handler always uses jinja to render cloudformation therefore this problem is probably related to how jinja processes the file and is probably related to https://github.com/orgs/Sceptre/discussions/1238 and #1380

[zaro0508]

ohh oops, i'm wrong. there are different paths for jinja and non-jinja processing but anyways that's the code that does the processing.

[wieshka]

Pre-processing the very same JSON template with JQ yields a template which works with Sceptre, but changes (diff) in the file are only formatting related such as identation, whitespace between "key": "value" instead if "key":"value"

[wieshka]

Stripped template to bare minimum and I am still able to reproduce it:

{
   "AWSTemplateFormatVersion":"2010-09-09",
   "Metadata": {
      "cfn-lint": {
          "config": {
              "ignore_checks": ["E2531"]
          }
      }
   },
   "Description":"Configure basic settings required by Security (v.4.3.1)",
   "Resources":{
      "ctrail":{
         "Type":"AWS::CloudTrail::Trail",
         "Properties":{
            "EnableLogFileValidation":true,
            "IncludeGlobalServiceEvents":true,
            "IsLogging":true,
            "IsMultiRegionTrail":true,
            "S3BucketName":"aaaaa"
         }
      }
   }
 }

[wieshka]

My further investigation (monkey patching) leads to findings, that if, I extend template_handler.http handle() with respective checks - is file JSON or YAML, and load them into dict with respective library per syntax, and then dump them back as string for on return, issue gets resolved.

So that, could be potential fix to address this, unless somebody has better ideas or actually can figure out why it falls apart with sample provided above which is a valid JSON.

Essentially this mimics my previous observation that pre-procesing content with commonly used JQ resolves the issue as well.

[wieshka]

Further investigation reveals that, if JSON template, fetched by http handler, contains mixed use of spaces and tabs for identation, it will further fail with:

botocore.exceptions.ClientError: An error occurred (ValidationError) when calling the CreateStack operation: Template format error: YAML not well-formed. (line 23, column 1).

[zaro0508]

@wieshka i suggest you use pre-commit hooks to pre format your files. We use remove-tabs, end-of-file-fixer, and trailing-whitespace

[iainelder]

@wieshka provided this sample template in the Sceptre Slack channel.

https://gist.githubusercontent.com/wieshka/91d37fe765b2a1189fb3d80dff71f9ed/raw/33f1153c9d05ac2ef290d5187f203de38e1bf725/sample.json

It mixes tabs and spaces for indentation. @wieshka says it invokes the reported error. (I haven't tried to reproduce it yet.)

---

In general I would avoid mixing tabs and spaces for indentation if you can avoid it. @zaro0508 's pre-commit hooks can help with that.

But I still don't understand why it would provoke this error in CloudFormation.

I want to dig deeper into this. Some questions:

• Does Sceptre always treat the JSON input as YAML, with or without explicit conversion? YAML is supposed to be a superset of JSON, but I'm sure there are inconsistencies.
• Is a YAML file even valid with mixed indentation?
• Is a JSON file even valid with mixed indentation?

[zaro0508]

closing this issue due to lack of response from the reporter