Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve sceptre.stack.Stack.__repr__() #570

Closed
1oglop1 opened this issue Dec 26, 2018 · 6 comments
Closed

Improve sceptre.stack.Stack.__repr__() #570

1oglop1 opened this issue Dec 26, 2018 · 6 comments

Comments

@1oglop1
Copy link
Contributor

1oglop1 commented Dec 26, 2018

Hi,
during the poc for #553 I found out that the __repr__ is incorrect.
The __repr__ method should ideally resolve into initialised object.
And cannot be directly transformed into a python object due to problem that stack_group_config is casted into str and once repr is printed it creates sceptre_user_date='{'Content'}'
same problem applies to sceptre_user_data

c98755a#r31781278

I fixed the repr, if you paste this repr into interpreter it will return

sceptre.stack.Stack(name='lab/iam-managed-policies', project_code='CR-Vault',
                    template_path='project/templates/iam_managed_policies.py',
                    region='eu-west-1', template_bucket_name='None', template_key_prefix='None',
                    required_version='None', profile='None', sceptre_user_data={'ManagedPolicies': {'ConsulPolicy': {
        'PolicyDocument': {'Version': '2012-10-17',
                           'Statement': [{'Effect': 'Allow', 'Action': ['ec2:Describe*'], 'Resource': ['*']},
                                         {'Effect': 'Allow', 'Action': ['s3:ListBucket*'],
                                          'Resource': ['arn:aws:s3:::cr-lab-jan']},
                                         {'Effect': 'Allow', 'Action': ['s3:PutObject', 's3:GetObject'],
                                          'Resource': ['arn:aws:s3:::lab-jan/consul-snapshot/*']}]},
        'Description': 'This is the Consul IAM policy'}, 'VaultPolicy': {'PolicyDocument': {'Version': '2012-10-17',
                                                                                            'Statement': [
                                                                                                {'Effect': 'Allow',
                                                                                                 'Action': [
                                                                                                     'ec2:Describe*'],
                                                                                                 'Resource': ['*']},
                                                                                                {'Effect': 'Allow',
                                                                                                 'Action': [
                                                                                                     'dynamodb:CreateTable',
                                                                                                     'dynamodb:BatchWriteItem',
                                                                                                     'dynamodb:UpdateItem',
                                                                                                     'dynamodb:BatchGetItem',
                                                                                                     'dynamodb:DescribeTable',
                                                                                                     'dynamodb:GetItem',
                                                                                                     'dynamodb:ListTables',
                                                                                                     'dynamodb:Query',
                                                                                                     'dynamodb:Scan',
                                                                                                     'dynamodb:DescribeReservedCapacity',
                                                                                                     'dynamodb:DescribeReservedCapacityOfferings',
                                                                                                     'dynamodb:ListTagsOfResource',
                                                                                                     'dynamodb:DescribeTimeToLive',
                                                                                                     'dynamodb:DescribeLimits'],
                                                                                                 'Resource': [
                                                                                                     'arn:aws:dynamodb:*:*:table/vault-data']}]},
                                                                         'Description': 'This is the Vault IAM policy'}}},
                    parameters='{}', hooks='{}', s3_details='None', dependencies='[]', role_arn='None',
                    protected='False', tags='{}', external_name='lab-iam-managed-policies', notifications='[]',
                    on_failure='None', stack_timeout='0',
                    stack_group_config={'consul_backup_prefix': 'consul-backup', 'consul_backup_bucket': 'cr-lab-jan',
                                        'instance_ssh_key_pair_name': 'access-mgmt',
                                        'dynamodb_vault_table': 'vault-data', 'account_id': '333639628900',
                                        }
                    )

It will return.

sceptre.stack.Stack(name='lab/iam-managed-policies', project_code='CR-Vault', template_path='project/templates/iam_managed_policies.py', region='eu-west-1', template_bucket_name='None', template_key_prefix='None', required_version='None', profile='None', sceptre_user_data='{'ManagedPolicies': {'ConsulPolicy': {'PolicyDocument': {'Version': '2012-10-17', 'Statement': [{'Effect': 'Allow', 'Action': ['ec2:Describe*'], 'Resource': ['*']}, {'Effect': 'Allow', 'Action': ['s3:ListBucket*'], 'Resource': ['arn:aws:s3:::cr-lab-jan']}, {'Effect': 'Allow', 'Action': ['s3:PutObject', 's3:GetObject'], 'Resource': ['arn:aws:s3:::lab-jan/consul-snapshot/*']}]}, 'Description': 'This is the Consul IAM policy'}, 'VaultPolicy': {'PolicyDocument': {'Version': '2012-10-17', 'Statement': [{'Effect': 'Allow', 'Action': ['ec2:Describe*'], 'Resource': ['*']}, {'Effect': 'Allow', 'Action': ['dynamodb:CreateTable', 'dynamodb:BatchWriteItem', 'dynamodb:UpdateItem', 'dynamodb:BatchGetItem', 'dynamodb:DescribeTable', 'dynamodb:GetItem', 'dynamodb:ListTables', 'dynamodb:Query', 'dynamodb:Scan', 'dynamodb:DescribeReservedCapacity', 'dynamodb:DescribeReservedCapacityOfferings', 'dynamodb:ListTagsOfResource', 'dynamodb:DescribeTimeToLive', 'dynamodb:DescribeLimits'], 'Resource': ['arn:aws:dynamodb:*:*:table/vault-data']}]}, 'Description': 'This is the Vault IAM policy'}}}', parameters='{}', hooks='{}', s3_details='None', dependencies='[]', role_arn='None', protected='False', tags='{}', external_name='lab-iam-managed-policies', notifications='[]', on_failure='None', stack_timeout='0', stack_group_config='{'consul_backup_prefix': 'consul-backup', 'consul_backup_bucket': 'cr-lab-jan', 'instance_ssh_key_pair_name': 'access-mgmt', 'dynamodb_vault_table': 'vault-data', 'account_id': '333639628900'}')

If you paste this again, you get syntax errors.
first at: sceptre_user_data='{'ManagedPolicies':
then: stack_group_config='{'consul_backup_prefix':

Fix should be easy just to remove single quotes around all objects in sceptre.stack.Stack.__repr__() and let python serialise values correctly.

EDIT:
moreover entire repr is in conflict with docstring, where docstring says the type is dict, repr retruns '{}' which is string.

@1oglop1
Copy link
Contributor Author

1oglop1 commented Dec 26, 2018

To tackle this is usually better to use some automatic repr creation either something provided via attrs or fn similar to this https://github.com/alexprengere/reprmixin/blob/master/reprmixin.py

@1oglop1
Copy link
Contributor Author

1oglop1 commented Dec 26, 2018

Another part of the issue is that sceptre has incorrectly handled API (I know the API is not meant to be used directly) which is also in conflict with current repr implementation

import sceptre
dir(sceptre)
['NullHandler', '__author__', '__builtins__', '__cached__', '__doc__', '__email__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', '__version__', 'logging']

So there is no way to get sceptre.stack.Stack as it's not accessible from within any ___init__.py

to fix this either use __all__ or at least import sceptre.stack in sceptre/__init__.py

@ngfgrant
Copy link
Contributor

Not taken too much time to look into this although on first pass, when I call repr() on stack I get something like the following:

ipdb> stack.__repr__()

"sceptre.stack.Stack(name='bundle/b/y', project_code='sceptre-example', template_path='/path-to-sceptre/sceptre-examples/simple/templates/vpc.json', region='eu-west-2', template_bucket_name='None', template_key_prefix='None', required_version='None', profile='None', sceptre_user_data='{}', parameters='{'CidrBlock': '10.0.0.0/16', 'VpcId': 'vpc-b-13'}', hooks='{}', s3_details='None', dependencies='['bundle/c/z.yaml']', role_arn='None', protected='False', tags='{}', external_name='sceptre-example-bundle-b-y', notifications='[]', on_failure='None', stack_timeout='0', stack_group_config='{}')"

I think this is an alright representation of the object?

As per the docs there is no promise of repr being the exact object representation.

object.repr(self)

Called by the repr() built-in function to compute the “official” string representation of an object. If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment). If this is not possible, a string of the form <...some useful description...> should be returned. The return value must be a string object. If a class defines repr() but not str(), then repr() is also used when an “informal” string representation of instances of that class is required.

We can amend the values for sceptre_user_data and stack_group_config so that they are not strings, although I am not sure you can say repr "doesn't work"?

@1oglop1
Copy link
Contributor Author

1oglop1 commented Dec 26, 2018

No it's not but it's way easier to debug when it is.

If at all possible, this should look like a valid Python expression that could be used to recreate an object with the same value (given an appropriate environment).

Okay simply try this with your repr

your_repr = "sceptre.stack.Stack(name='bundle/b/y', project_code='sceptre-example', template_path='/path-to-sceptre/sceptre-examples/simple/templates/vpc.json', region='eu-west-2', template_bucket_name='None', template_key_prefix='None', required_version='None', profile='None', sceptre_user_data='{}', parameters='{'CidrBlock': '10.0.0.0/16', 'VpcId': 'vpc-b-13'}', hooks='{}', s3_details='None', dependencies='['bundle/c/z.yaml']', role_arn='None', protected='False', tags='{}', external_name='sceptre-example-bundle-b-y', notifications='[]', on_failure='None', stack_timeout='0', stack_group_config='{}')"

eval(your_repr)


Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "<string>", line 1
    sceptre.stack.Stack(name='bundle/b/y', project_code='sceptre-example', template_path='/path-to-sceptre/sceptre-examples/simple/templates/vpc.json', region='eu-west-2', template_bucket_name='None', template_key_prefix='None', required_version='None', profile='None', sceptre_user_data='{}', parameters='{'CidrBlock': '10.0.0.0/16', 'VpcId': 'vpc-b-13'}', hooks='{}', s3_details='None', dependencies='['bundle/c/z.yaml']', role_arn='None', protected='False', tags='{}', external_name='sceptre-example-bundle-b-y', notifications='[]', on_failure='None', stack_timeout='0', stack_group_config='{}')
                                                                                                                                                                                                                                                                                                                            ^
SyntaxError: invalid syntax

my fixed repr did not return syntax error.
so it would be better either to return just Stack.__dict__ or string which can be evaluated into obj.

@ngfgrant ngfgrant changed the title sceptre.stack.Stack.__repr__() does not work out of the box Improve sceptre.stack.Stack.__repr__() Dec 26, 2018
@ngfgrant
Copy link
Contributor

For this issue I am fine with either leaving it how it is or you can put in a PR for removing the quotes around the dicts in Stack.

Can I ask if you are planning on using eval() in your solution to #553 or if you mearly stumbled upon the issue when playing around? You should be able to get pretty much all the useful debugging info from SceptrePlan and SceptreContext?

@1oglop1
Copy link
Contributor Author

1oglop1 commented Dec 28, 2018

no I'm not planning to use any eval but it's one of the repr tests you could use.
assert isinstace(eval(repr(object)), ObjClass)

This comes to my earlier point (~3 months) about 100% test coverage .. it does not matter if coverage is 100%, the quality of tests matter.

ngfgrant added a commit that referenced this issue Jan 9, 2019
Previously, dict and list types were contained within strings in
sceptre.stack.Stack.__repr__(). This commit removes the strings so that
a stack.__repr__() can be used to create a stack using eval().
ngfgrant added a commit that referenced this issue Jan 9, 2019
Previously, dict and list types were contained within strings in
sceptre.stack.Stack.__repr__(). This commit removes the strings so that
a stack.__repr__() can be used to create a stack using eval().
kierandoonan added a commit that referenced this issue Jan 10, 2019
ngfgrant added a commit that referenced this issue Jan 10, 2019
Previously, dict and list types were contained within strings in
sceptre.stack.Stack.__repr__(). This commit removes the strings so that
a stack.__repr__() can be used to create a stack using eval().

In order to properly compare stacks a custom `__eq__` and therefore
`__hash__` implementations were added to Stack. Adding a correct `__eq__`
method to Stack allowed testing of Stack objects that were created from
`eval()` of the stack `__repr__`.
ngfgrant added a commit that referenced this issue Jan 10, 2019
Previously, dict and list types were contained within strings in
sceptre.stack.Stack.__repr__(). This commit removes the strings so that
a stack.__repr__() can be used to create a stack using eval().

In order to properly compare stacks a custom `__eq__` and therefore
`__hash__` implementations were added to Stack. Adding a correct `__eq__`
method to Stack allowed testing of Stack objects that were created from
`eval()` of the stack `__repr__`.
ngfgrant added a commit that referenced this issue Jan 16, 2019
Previously, dict and list types were contained within strings in
sceptre.stack.Stack.__repr__(). This commit removes the strings so that
a stack.__repr__() can be used to create a stack using eval().

In order to properly compare stacks a custom `__eq__` and therefore
`__hash__` implementations were added to Stack. Adding a correct `__eq__`
method to Stack allowed testing of Stack objects that were created from
`eval()` of the stack `__repr__`.
alex-harvey-z3q added a commit that referenced this issue Aug 30, 2024
Way back in 2018, the original team were adding the debug command, and in
that context set about to improve the output in some debug circumstances
during inspection of Sceptre's internal objects.

In response to #570, a custom
`__eq__` method was added in df9fc20.

Based on what can be seen of the original implementation of the custom
`__eq__`, it has not been maintained in a long time, other than changes
to it that appeared to cause issues for the current maintainers.

--

A recent change to improve the output during cyclical dependency errors
has introduced a call to `nx.find_cycle`. This function explicitly
searches for a cycle in the graph. This appears to resolve in code
testing for equality of nodes and thus calls to our custom `__eq__`
function.

The `__eq__` method however has been technically broken ever since
`template_path` was made optional. This then results in `__eq__`
failing. This fails as the code to indicate that `template_path` is
deprecrated is then executed, which will fail if the setting has not
really been defined.

--

This PR proposes to simply remove `__eq__` since it is believed that it
is no longer in use by anything. It has been more than 2 years since
`template_path` was deprecrated and made optional, and the fact of this
bug having never been observed until now is good evidence that it is not
used or needed.
alex-harvey-z3q added a commit that referenced this issue Aug 30, 2024
Way back in 2018, the original team were adding the debug command, and in that context set about to improve the output in some debug circumstances during inspection of Sceptre's internal objects.

In response to #570, a custom __eq__ method was added in df9fc20.

Based on what can be seen of the original implementation of the custom __eq__, it has not been maintained in a long time, other than changes to it that appeared forced on the current maintainers.

A recent change 993ef09 was done to improve the output during cyclical dependency errors. This change introduced a call to nx.find_cycle, a function that explicitly searches for a cycle in the graph. This appears to result in code testing for the equality of nodes in the graph and thus calls to the custom __eq__ method.

The __eq__ method however has been technically broken ever since template_path was made optional. This then results in __eq__ failing. This fails as the code to announce the deprecation of template_path is then executed, which fails if the deprecated setting is not actually in use.

This PR proposes to simply remove __eq__ since it is believed that it is no longer in use by anything. It has been more than 2 years since template_path was deprecrated and made optional, and the fact that this bug has not surfaced until now is good evidence that the code is not otherwise used or needed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants