Make all test properties dynamic, and support much easier ways to set them

[svanoort]

As a user of PyRestTest, I'd like to be able to set all test properties dynamically so I don't repeat myself. It would also be nice to make use of generators and variable simpler, and support raw values (not just strings for templates).

Nice to have: extensible templating, using something such as Jinja2.

Suggested syntax:

- variable_value: {variable: 'myVariableName'}  # Raw value of variable is used
- url_encoded: {url_encode: 'myVariableName'}   #  Url-encoded value of variable
- expression: {expression: 'string of text'}  # Some sort of general expression language (later)
- file_content: {file: 'myFile'}  # File is a path
- standard: {template: 'template $value in string'}  # old form
- generator_property: {generator: 'myGeneratorName'}  # Uses name of generator to store value
- special_text: {jinja2: 'value_template_whee'}  # Custom template text

Better/shorter syntaxes for this (I.E. look at ansible)

• Easy Solution: provide a pipe-like syntax for dynamic value resolution.
  • Ex: 'file | url_encode | template | stuff.value' as a single string
  • Might roll up this one nicely: Easier ability to submit form data by POST #115
  • Internally this pipeline is parsed into a Pipeline object with a series of steps.
  • Pipeline object is easy to test, each step can be tested individually, and pipeline coupling between steps is easy to test
  • Each step has a simple API:

StepName(StepClass):
 name = myName   # Registered name, can be a collection if you give aliases
 allowed_inputs = None   # Maybe add this, checks that input types match
 allowed_outputs = None  # Maybe add this, allows you to type-check outputs/inputs in pipe from each other
 evaluate(input, context, *args, **kwargs): 
  """ Evaluates the step, using the input and returning an object back, usable in other pipelines """

• Type checking may be provided later, to allow catching issues in pipeline before execution of test.

Challenges:

• what if we have multiple items? Like, a template and a file value? Which is used?
  • This might be the biggest problem! Also the syntax is slightly clunky with a dictionary type
    • Solution: fail on parsing with an error that it is ambiguous.
• nesting: template + file + jinja? (should be supported, but need test just like Contexts)
• dynamic-to-dynamic references: is it evaluated up-front when first encountered in tests, i.e. define variable dynamically then evaluate, or lazily every time?

Work Items:

• Depends on parsing refactoring to be easy to use
• Needs to store the dynamic portion and then replace every getter call with a property?
• Caching of results if not changing?
• Architecture: use properties/proxies fed by the context?

For nice-to-have:

• registry approach to parsing/looking up variable data sources

Example:

• Content module, used for request bodies, etc
• Step 1: general architectural changes to allow for this use globally and register a few basic transformations (file & template for content)

Work Items

• Extend the testconfig/macros objects to do dynamic resolution using context
• Remove shitty templating feature
• Create special property-generating parsers for the dynamic elements that use context/with items to find values

[svanoort]

@mariusmagureanu I feel like you might have some good ideas on this one -- any notions?

[svanoort]

To support this, we need to extend the content-handler into a 'resolver' class that deals with more complex variable use/binding I think. It also needs to offer pythonic APIs, because that is likely the bit that will be modified the soonest.

[svanoort]

Validators need to support this for expected values, so you can use actual variables in your expected value tests, per #135 - without the nasty templates hack :(

Tentatively rescheduling this as part of the 1.8.0 milestone, but something will need to be cut -- hard decisions there.

[svanoort]

Needs at least a few basic transformation options, per #159

Probably something like url_encode would be highly desirable.

This might also cover a lot of the cases in #115

[svanoort]

Gathering implementation thoughts:

This is going to be kind of complex and might take a couple prototypes to get right.

Probably ties with #75 to some extent.

In ANY case we're going to PURGE WITH FIRE the horrors of the current properties methods (with get/set/realize per field) and save grief by embedding it into our Test/Benchmark class instance or somehow using a local context.

Solution: use of with-statement to embed the Context for transparent use by class?

Kind of a hacky example for this

#!/usr/bin/env python
# Hacky example

#!/usr/bin/env python

class contextbinding:
  bindings = None
  original = None

  def __init__(self, key, value):
    self.bindings = dict()
    self.bindings[key] = value

  def __enter__(self):
    try:
      self.original = bound
    except NameError:
      pass
    global bound
    bound = self.bindings

  def __exit__(self, etype, value, traceback):
    global bound
    bound = self.original
    pass

def get_bound(variable_name):
  try:
    return bound[variable_name]
  except NameError:
    pass
  except KeyError:
    pass
  try:
    return self.bound[variable_name]
  except NameError:
    pass
  except KeyError:
    pass
  print("Variable not declared and needs binding: {0}".format(variable_name))
  return None


# Demonstrate bindings option using a context?
keyname = 'original val'
print(get_bound('keyname'))

bound = {'keyname': 'awesome'}
print(get_bound('keyname'))

with contextbinding('keyname','goober') as lolbindingslol:
  print(get_bound('keyname'))
print(get_bound('keyname'))

Simplest approach:

• Each canonical test gets a clone with dynamic fields substituted, just like [calling the current realize method[(https://github.com/svanoort/pyresttest/blob/master/pyresttest/tests.py#L241)
  • Easy to implement I think (create properties for each item, with getters that use an internally bound context at runtime to generate result, then do a lookup for each field this way)
  • Ex: when parsing with a body from generator, a property is created instead of setting the value. Property does string.template(_body).safe_substitute(context.get_variables()) or similar.
  • Bad for benchmarks because they would require creating a copy every time. But for these if we move the execution into the benchmark class itself, it can do some internal optimization here!
• Full-dynamic property access, using bound context:
  • Canonical test/benchmark steps are copied for runtime use and reused - the context is bound in, and properties are used similar to above, however there is no full 'realize' step applied ahead of time. Each property access dynamically evaluates, and benchmark may be called with the same context multiple times.

Harder approach:

• Try to anticipate the sequence of values, I.E. pre-generate generator results?

Coupling with parsing:

• Every object with parse will need some sort of parsing class variable, which ties to this class to do the dynamic variable configuration, this can also be exposed by API

Ex something like:

TestConfig():
  myparser = ParserClass(my_allowed_options)

def parse(config):
  for option in config:
     parserclass.parse(self_to_set_up, config_option)   # handles dynamic stuffs

def configure():
  pass  # Implementation & API TBD, this would be the pythonic/fluent API though.