Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Key/Value backend #865

Closed
wants to merge 81 commits into from
Closed

Key/Value backend #865

wants to merge 81 commits into from

Conversation

@mogorman
Copy link

@mogorman mogorman commented Feb 4, 2018

This is just the same as #624 I will make the other hundred refrence changes needed but i just wanted to get this started.

There are quite some ideas in the issue tracker about how we could improve nixops radically.

I would like to start out with a feasible iterative plan. As far as I can see, only very trivial SQL operations are every executed against the sqlite3 database; the main use case is to use transactions to support rollback in the case of exceptions. This should not be to hard to implement ourselves in memory, in combination with a key/value server that supports locking certain keys.

  • Refactor the statefile and deployment lock into a specific local implementation
  • Introduce different URI schema's for future backends
  • Refactor all the sqlite3 queries into concrete operations implemented in file.Statefile
  • Deployment
  • Resource
  • VirtualBox
  • None backend done, needs to be verified
  • GCE done, needs to be verified
  • AWS done, needs to be verified
  • Hetzner done, needs to be verified
  • Implement a local json file backend
  • Fix unit tests
moretea added 30 commits Mar 13, 2017
Needed to faclitate multiple backends (like sqlite3/other remote database)
For now it's just the underlying DB, but it resembles something that
can roll back the current transaction.
@mogorman
Copy link
Author

@mogorman mogorman commented Feb 19, 2018

@AmineChikhaoui I believe i fixed these unit test failures with my last round of commits, they fail for me stil as I don't have aws keys but they do correctly update the attrs and get past the failure point you posted.

@tomberek Your issue is coming from nix-shell over escaping things. probably because you were in the dev-shell and then executed another deeper nix-shell. can you try installing this fork of nixops and running without the nested shells? this error does not happen to me when i used <> directly

{
  "schemaVersion": 0, 
  "deployments": {
    "8f9506b0-153e-11e8-839e-024290535936": {
      "attributes": {
        "nixExprs": "[\"<trivial.nix>\", \"<trivial-vbox.nix>\"]", 
        "name": "test", 
        "configsPath": "/nix/store/dm9a1cixp8w7mylzv6vhpwv7nj0i2kqr-nixops-machines", 
        "description": "Web server"
      }, 
      "resources": {}
    }
  }
}
@AmineChikhaoui
Copy link
Member

@AmineChikhaoui AmineChikhaoui commented Feb 19, 2018

@mogorman Thanks, so the previous errors are gone but I still see some errors during the deployment tear down in the unit tests, can't post them now because it exposes some resources IDs and AWS debug output but I'll re-run it later this week and look at the errors.

@mogorman
Copy link
Author

@mogorman mogorman commented Feb 23, 2018

@AmineChikhaoui did you ever get a chance to get me more info as to what is broken? or can anyone help me understand how to run the test framework better? the only errors i get when running tests are those caused by lack of aws gce etc credentials so i assume the other tests passed?

@andrewchambers
Copy link

@andrewchambers andrewchambers commented Mar 9, 2018

Could someone explain the rationale for this change? what is the point?

@mogorman
Copy link
Author

@mogorman mogorman commented Mar 9, 2018

the point of providing a plugable state backend is to make it easier for people to work with nixops. In the case of the json one, it makes it much easier to commit these things to git and share them among a team, in the future I want to make a mysql/postgresql backend as well so that I can manage the state there. Others have requested a backend using s3 and simpledb. this work makes it easier to do all these things.

@moretea
Copy link
Contributor

@moretea moretea commented Mar 9, 2018

Having a json file is the starting point to prove that we can use any K/V store. Once that's done, adding e.g. mysql, or S3 + DynamoDB should just work fine.

@gleber-da
Copy link

@gleber-da gleber-da commented Apr 26, 2018

👍 to these changes. These would help us share the workload of managing our deployments in the team.

@nh2
Copy link
Contributor

@nh2 nh2 commented May 27, 2018

What's the state currently? Do you only need testers for the 3 remaining backends?

@moretea
Copy link
Contributor

@moretea moretea commented May 29, 2018

@nh2 talking to me? And which backends are you referring to?

@nh2
Copy link
Contributor

@nh2 nh2 commented May 29, 2018

To whomever would like this PR in, really. I was referring to

  • None backend done, needs to be verified
  • GCE done, needs to be verified
  • AWS done, needs to be verified
  • Hetzner done, needs to be verified

in the issue description.

@srghma
Copy link
Contributor

@srghma srghma commented Jun 9, 2018

  1. aws test python2 tests.py tests.functional.test_backups didn't succeed for me https://pastebin.com/ZmqNeZHW
  2. all tests are using sqlite state file, there is no tests that use json as I understand
@srghma
Copy link
Contributor

@srghma srghma commented Jun 9, 2018

@mogorman I've made a pr mogorman#3


def _delete_deployment(self, deployment_uuid):
"""NOTE: This is UNSAFE, it's guarded in nixops/deployment.py. Do not call this function except from there!"""
self.__db.execute("delete from Deployments where uuid = ?", (deployment_uuid,))

This comment has been minimized.

@srghma

srghma Jun 10, 2018
Contributor

lol, how this line got here? its causing errors

[nix-shell:~/projects/nixops]$ python2 tests.py tests.functional.test_ec2_backups
ssh-security-group> creating EC2 security group ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group’...
ssh-security-group> adding new rules to EC2 security group ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group’...
my-key-pair.......> uploading EC2 key pair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’...
machine...........> creating EC2 instance (AMI ‘ami-d464cba9’, type ‘c4.large’, region ‘us-east-1’)...
warning: are you sure you want to destroy EC2 machine ‘machine’? (y/N) y
machine...........> destroying EC2 machine... [shutting-down] [shutting-down] [terminated]
warning: are you sure you want to destroy keypair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’? (y/N) y
my-key-pair.......> deleting EC2 key pair ‘charon-15e3710a-6cce-11e8-8e29-02424049f177-my-key-pair’...
ssh-security-group> deleting EC2 security group `charon-15e3710a-6cce-11e8-8e29-02424049f177-ssh-security-group' ID `sg-6bbd5620'...
Exception in thread Thread-4:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 47, in destroy_deployments
    deployment.delete()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 189, in delete
    self._state._delete_deployment(self.uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 201, in _delete_deployment
    self.__db.execute("delete from Deployments where uuid = ?", (deployment_uuid,))
AttributeError: 'JsonFile' object has no attribute '_JsonFile__db'

warning: not all deployments have been destroyed; some resources may still exist!
E
======================================================================
ERROR: tests.functional.test_ec2_backups.test_ec2_backups('json', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_ebs.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'], '')
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda>
    nose_func = wraps(func)(lambda *args: func(*args))
  File "/home/srghma/projects/nixops/tests/functional/test_ec2_backups/__init__.py", line 30, in test_ec2_backups
    deployment.deploy()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 978, in run_with_notify
    f()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 922, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/home/srghma/projects/nixops/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 895, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/home/srghma/projects/nixops/nixops/backends/ec2.py", line 1012, in create
    self.update_tags(self.vm_id, user_tags=common_tags, check=check)
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 45, in update_tags
    self.update_tags_using(updater, user_tags=user_tags, check=check)
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 33, in update_tags_using
    tags.update(self.get_common_tags())
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 22, in get_common_tags
    'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)}
AttributeError: TransactionalJsonFile instance has no attribute 'db_file'
-------------------- >> begin captured logging << --------------------
boto: DEBUG: Using access key provided by client.
boto: DEBUG: Using secret key provided by client.
self._rollback()
else:
self._commit()
fcntl.flock(self._lock_file, fcntl.LOCK_UN)

This comment has been minimized.

@srghma

srghma Jun 10, 2018
Contributor

aren't we should remove lock file? @mogorman

@@ -19,7 +19,7 @@ def _retry(self, fun, **kwargs):
def get_common_tags(self):
tags = {'CharonNetworkUUID': self.depl.uuid,
'CharonMachineName': self.name,
'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._db.db_file)}
'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)}

This comment has been minimized.

@srghma

srghma Jun 10, 2018
Contributor

I'm getting this error

Traceback (most recent call last):
  File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda>
    nose_func = wraps(func)(lambda *args: func(*args))
  File "/home/srghma/projects/nixops/tests/functional/test_ec2_backups/__init__.py", line 30, in test_ec2_backups
    deployment.deploy()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in deploy
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 978, in run_with_notify
    f()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 989, in <lambda>
    self.run_with_notify('deploy', lambda: self._deploy(**kwargs))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 922, in _deploy
    nixops.parallel.run_tasks(nr_workers=-1, tasks=self.active_resources.itervalues(), worker_fun=worker)
  File "/home/srghma/projects/nixops/nixops/parallel.py", line 44, in thread_fun
    result_queue.put((worker_fun(t), None, t.name))
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 895, in worker
    r.create(self.definitions[r.name], check=check, allow_reboot=allow_reboot, allow_recreate=allow_recreate)
  File "/home/srghma/projects/nixops/nixops/backends/ec2.py", line 1012, in create
    self.update_tags(self.vm_id, user_tags=common_tags, check=check)
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 45, in update_tags
    self.update_tags_using(updater, user_tags=user_tags, check=check)
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 33, in update_tags_using
    tags.update(self.get_common_tags())
  File "/home/srghma/projects/nixops/nixops/resources/ec2_common.py", line 22, in get_common_tags
    'CharonStateFile': "{0}@{1}:{2}".format(getpass.getuser(), socket.gethostname(), self.depl._state.db.db_file)}
AttributeError: TransactionalJsonFile instance has no attribute 'db_file'

So I will rename self._db_file for json state to self.db_file

@srghma
Copy link
Contributor

@srghma srghma commented Jun 11, 2018

@mogorman how could one convert sqlite store file to json?

@srghma srghma mentioned this pull request Jun 14, 2018
21 of 22 tasks complete

def clone_deployment(self, deployment_uuid):
with self.db:
if not uuid:

This comment has been minimized.

@srghma

srghma Jun 16, 2018
Contributor

this line was causing errors

[nix-shell:~/projects/nixops]$ python2 tests.py tests.functional.test_cloning_clones -vv
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$']
nose.selector: INFO: /home/srghma/projects/nixops/dev-shell is executable; skipped
nose.selector: INFO: /home/srghma/projects/nixops/coverage-tests.py is executable; skipped
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$']
tests.functional.test_cloning_clones.test_cloning_clones('json', ('ec2', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix'])) ... deployment ‘7b0585d1-7144-11e8-95a9-02421304dbb8’ destroyed
ERROR

======================================================================
ERROR: tests.functional.test_cloning_clones.test_cloning_clones('json', ('ec2', ['/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/logical_base.nix', '/home/srghma/projects/nixops/tests/functional/shared/nix_expressions/ec2_base.nix']))
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/qyqsfqjpksrvaa387yk4y5bgxnslfyx8-python2.7-nose-1.3.7/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
    self.test(*self.arg)
  File "/nix/store/kwq9a1bjhc3w1672khipz85qrwpxj5lv-python2.7-parameterized-0.6.1/lib/python2.7/site-packages/parameterized/parameterized.py", line 280, in <lambda>
    nose_func = wraps(func)(lambda *args: func(*args))
  File "/home/srghma/projects/nixops/tests/functional/test_cloning_clones/__init__.py", line 63, in test_cloning_clones
    deployment2 = deployment.clone()
  File "/home/srghma/projects/nixops/nixops/deployment.py", line 164, in clone
    return self._state.clone_deployment(self.uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 208, in clone_deployment
    if not uuid:
UnboundLocalError: local variable 'uuid' referenced before assignment

----------------------------------------------------------------------
Ran 1 test in 0.098s

FAILED (errors=1)

fixed with a681561


def open_deployment(self, uuid=None):
"""Open an existing deployment."""
deployment = self._find_deployment(uuid=uuid)

This comment has been minimized.

@srghma

srghma Jun 16, 2018
Contributor

I was getting error here


[nix-shell:~/projects/nixops]$ rm -rfd tests/state_files && python2 tests.py tests.functional.test_query_deployments -vv
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$']
nose.selector: INFO: /home/srghma/projects/nixops/dev-shell is executable; skipped
nose.selector: INFO: /home/srghma/projects/nixops/coverage-tests.py is executable; skipped
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
nose.config: INFO: Excluding tests matching ['^coverage-tests\\.py$']
tests.functional.test_query_deployments.test_query_deployments('json',) ... Exception in thread Thread-9:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Exception in thread Thread-10:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

Exception in thread Thread-4:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Exception in thread Thread-8:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
Exception in thread Thread-5:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded


Exception in thread Thread-7:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded


Exception in thread Thread-6:
Traceback (most recent call last):
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/home/srghma/projects/nixops/tests/functional/shared/using_state_file.py", line 31, in destroy_deployments
    deployment = state.open_deployment(uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 184, in open_deployment
    deployment = self._find_deployment(uuid=uuid)
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 156, in _find_deployment
    all_deployments = self.db.read()["deployments"]
  File "/home/srghma/projects/nixops/nixops/state/json_file.py", line 49, in read
    return json.load(f)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 291, in load
    **kw)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/__init__.py", line 339, in loads
    return _default_decoder.decode(s)
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 364, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/nix/store/wj95m7mnglqih1hhwyywchl0bgcldhad-python-2.7.15/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded

deployment ‘a93aca6b-715e-11e8-9f15-0242f1563f79’ destroyed
deployment ‘a93a4770-715e-11e8-b529-0242f1563f79’ destroyed
deployment ‘a93a743d-715e-11e8-87dc-0242f1563f79’ destroyed
warning: not all deployments have been destroyed; some resources may still exist!
ok
tests.functional.test_query_deployments.test_query_deployments('nixops',) ... deployment ‘a9634280-715e-11e8-8997-0242f1563f79’ destroyed
deployment ‘a9620902-715e-11e8-b304-0242f1563f79’ destroyed
deployment ‘a960bc33-715e-11e8-a0db-0242f1563f79’ destroyed
deployment ‘a965e614-715e-11e8-8ca3-0242f1563f79’ destroyed
deployment ‘a95cd39e-715e-11e8-8113-0242f1563f79’ destroyed
deployment ‘a9672c5e-715e-11e8-92eb-0242f1563f79’ destroyed
deployment ‘a95e2f8a-715e-11e8-955f-0242f1563f79’ destroyed
deployment ‘a9687533-715e-11e8-988f-0242f1563f79’ destroyed
deployment ‘a9648ba1-715e-11e8-ae66-0242f1563f79’ destroyed
deployment ‘a95f74f8-715e-11e8-b731-0242f1563f79’ destroyed
ok

----------------------------------------------------------------------
Ran 2 tests in 0.607s

OK

[nix-shell:~/projects/nixops]$

fix a681561

self._state.set_resource_attrs(self.uuid, self.id, {key:value})

def __getitem__(self, key):
value = self._state.get_resource_attr(self.uuid, self.id, key)

This comment has been minimized.

@srghma

srghma Jun 16, 2018
Contributor

I get an error here https://pastebin.com/N8E90uke because of the inconsistency between value type:

  • sqlite always returns a string (though this string may contain json, number or boolean)
  • json state file returns already parsed value

This comment has been minimized.

@srghma

srghma Jun 16, 2018
Contributor

fix e1b9158

@tomberek
Copy link
Contributor

@tomberek tomberek commented Nov 4, 2018

What is needed for this? Do you need user reports and testing?

@srghma
Copy link
Contributor

@srghma srghma commented Nov 4, 2018

@tomberek hi, I need hertzner testing here #969 (let's track this issue here #1036)

@domenkozar
Copy link
Member

@domenkozar domenkozar commented Mar 26, 2020

See #1264

@grahamc
Copy link
Member

@grahamc grahamc commented Mar 26, 2020

Hello!

Thank you for this PR.

In the past several months, some major changes have taken place in
NixOps:

  1. Backends have been removed, preferring a plugin-based architecture.
    Here are some of them:

  2. NixOps Core has been updated to be Python 3 only, and at the
    same time, MyPy type hints have been added and are now strictly
    required during CI.

This is all accumulating in to what I hope will be a NixOps 2.0
release
. There is a tracking issue for that:
#1242 . It is possible that
more core changes will be made to NixOps for this release, with a
focus on simplifying NixOps core and making it easier to use and work
on.

My hope is that by adding types and more thorough automated testing,
it will be easier for contributors to make improvements, and for
contributions like this one to merge in the future.

However, because of the major changes, it has become likely that this
PR cannot merge right now as it is. The backlog of now-unmergable PRs
makes it hard to see which ones are being kept up to date.

If you would like to see this merge, please bring it up to date with
master and reopen it
. If the or mypy type checking fails, please
correct any issues and then reopen it. I will be looking primarily at
open PRs whose tests are all green.

Thank you again for the work you've done here, I am sorry to be
closing it now.

Graham

@grahamc grahamc closed this Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

You can’t perform that action at this time.