Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kadet inputs with plain output type and dict objects fail to compile #445

Closed
uberspot opened this issue Jan 2, 2020 · 3 comments
Closed
Assignees
Labels

Comments

@uberspot
Copy link
Contributor

uberspot commented Jan 2, 2020

Describe the bug/feature
When Kadet is used in conjunction with plain output type it fails to write the output to a file.
The issue is in CompiledFile.write as it assumes that data passed to it is strings but that's not always the case (especially if in Kadet where you can define the exact outputted object as a python dictionary).

Solution: Convert data explicitly to str in write() (PR coming soon).

To replicate:

Kapitan config:

  kapitan:
    compile:
      - output_path: ""
        input_type: kadet
        output_type: plain
        input_paths:
          - components/bla

Kadet init.py:

.....
def main():
    obj = kadet.BaseObj()
    inv = kadet.inventory()
    kubelib = kadet.load_from_search_paths("kubelib")

    data = {}

    obj.root['my_dir/{}'.format(file_name)] = d['raw']

    obj.root['my_other_dir/my_config.yml'] = kubelib.ConfigMap(
        name='some_config_map',
        labels={},
        data=data
    )

    return obj

Stacktrace:

Traceback (most recent call last):
  File "~/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "~/kapitan/kapitan/targets.py", line 387, in compile_target
    input_compiler.compile_obj(comp_obj, ext_vars, **kwargs)
  File "~/kapitan/kapitan/inputs/base.py", line 59, in compile_obj
    self.compile_input_path(input_path, comp_obj, ext_vars, **kwargs)
  File "~/kapitan/kapitan/inputs/base.py", line 74, in compile_input_path
    target_name=target_name, **kwargs)
  File "~/kapitan/kapitan/inputs/kadet.py", line 134, in compile_file
    fp.write(item_value)
  File "~/kapitan/kapitan/inputs/base.py", line 107, in write
    self.fp.write(self.revealer.compile_raw(data, target_name=target_name))
  File "~/kapitan/kapitan/refs/base.py", line 314, in compile_raw
    compiled = self.regex.sub(self._compile_replace_match_with_args(**kwargs), data)
TypeError: expected string or bytes-like object
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "~/kapitan/kapitan/targets.py", line 95, in compile_targets
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "~/kapitan/kapitan/targets.py", line 95, in <listcomp>
    [p.get() for p in pool.imap_unordered(worker, target_objs) if p]
  File "~/.pyenv/versions/3.7.1/lib/python3.7/multiprocessing/pool.py", line 774, in next
    raise value
TypeError: expected string or bytes-like object


expected string or bytes-like object
@uberspot uberspot added the bug label Jan 2, 2020
@uberspot uberspot self-assigned this Jan 2, 2020
uberspot added a commit that referenced this issue Jan 2, 2020
@uberspot uberspot changed the title Kadet inputs with plain output type and json files fails to compile Kadet inputs with plain output type and dict objects fail to compile Jan 2, 2020
uberspot added a commit that referenced this issue Jan 2, 2020
@ramaro
Copy link
Member

ramaro commented Jan 2, 2020

output_path should be set regardless of this problem, I think? Setting subdirs in the filenames seems to me to be the problem, as it not supported - maybe leave this up to the user? Also, might be worth leaving plain alone and perhaps create a new bytes output?

@uberspot
Copy link
Contributor Author

uberspot commented Jan 2, 2020

output_path should be set regardless of this problem, I think?

Correct, it's an edge case/hack. But that is not the source causing this issue.

Setting subdirs in the filenames seems to me to be the problem, as it not supported - maybe leave this up to the user?

Correct as well, but I'd argue Kadet is one of the more versatile input types we have now and it would be a shame to not support this behavior in only one output type.

Also, might be worth leaving plain alone and perhaps create a new bytes output?

How come? I'd imagine plain should output plain strings anyway. And that conversion is not done anywhere explicitly before this PR. So it's a bug waiting to happen (?).

A bytes output would also be a legitimate fix for this use case, but I still would expect my plain output to work with non-string inputs and not get interpreted as different types of objects internally.

uberspot added a commit that referenced this issue Jan 3, 2020
uberspot added a commit that referenced this issue Jan 6, 2020
uberspot added a commit that referenced this issue Jan 6, 2020
Fix issue #445: Explicitly create directories for all outputted CompiledFile objects.
@uberspot
Copy link
Contributor Author

uberspot commented Jan 6, 2020

Closing this as the initial bug of directories not being created has been fixed in #446.
The conversion to str() can be done on the Kadet side as well so this is out of scope. :)

@uberspot uberspot closed this as completed Jan 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants