Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Postgres support for Frappe #5919

Merged
merged 161 commits into from Sep 21, 2018
Merged

Conversation

rmehta
Copy link
Member

@rmehta rmehta commented Aug 4, 2018

To create a new site (with Postgres DB)

bench new-site <site-name> --db-type postgres

@rmehta rmehta changed the title Pg poc [wip] [proposal] Postgres support in Frappe Aug 4, 2018
@realvinay
Copy link
Contributor

@riteshshrv , please test this branch with our app/datasets and provide feedback.

@riteshshrv
Copy link
Contributor

riteshshrv commented Aug 7, 2018

@surajshetty3416 https://github.com/frappe/frappe/blob/develop/frappe/database.py#L56
is there a specific reason not to inherit base class from object ?
Ref: https://stackoverflow.com/questions/4015417/python-class-inherits-object

I faced a issue while trying to init a new site using postgres, here's the traceback

  File "/usr/lib64/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/utils/bench_helper.py", line 97, in <module>
    main()
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/utils/bench_helper.py", line 18, in main
    click.Group(commands=commands)(prog_name='bench')
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/ritesh/Projects-Active/frappe-crux/env/lib/python2.7/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/commands/site.py", line 33, in new_site
    db_type = db_type)
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/commands/site.py", line 66, in _new_site
    source_sql=source_sql,force=force, reinstall=reinstall, db_type=db_type)
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/installer.py", line 38, in install_db
    setup_database(force, verbose)
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/database/__init__.py", line 13, in setup_database
    return frappe.database.postgres.setup_db.setup_database(force, verbose)
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/database/postgres/setup_db.py", line 6, in setup_database
    root_conn.commit()
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/database/database.py", line 757, in commit
    self.sql("commit")
  File "/home/ritesh/Projects-Active/frappe-crux/apps/frappe/frappe/database/postgres/database.py", line 101, in sql
    return super(PostgresDatabase, self).sql(query, *args, **kwargs)
TypeError: super() argument 1 must be type, not classobj

adding object inheritance fixed it

@surajshetty3416
Copy link
Member

surajshetty3416 commented Aug 7, 2018

@riteshshrv The link you sent describes that it makes no difference to inherit from an object if you are using Python 3.
And since I was on Python 3 while development I never came across that issue.

Anyways, I'll extend it from an object as we need to support Python 2 as well.

Thanks for reporting.

@riteshshrv
Copy link
Contributor

@surajshetty3416 there seems to be some problem with "sessions";
i do not have complete information as there's no relevant traceback in both browser and terminal console but just letting you know in case if it might be helpful.
this does not happen if you login in Incognito!
image

creating a new doctype failed, is it just me or this part is still under development?

@surajshetty3416
Copy link
Member

surajshetty3416 commented Aug 14, 2018

@riteshshrv I have fixed Doctype Sessions not found issue... please pull the latest code and try again.

Also, will fix that new Doctype creation issue and let you know..

@riteshshrv
Copy link
Contributor

@surajshetty3416 i'm using python2.7 env and login is now broken in both regular and incognito

  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/app.py", line 53, in application
    init_request(request)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/app.py", line 115, in init_request
    frappe.local.http_request = frappe.auth.HTTPRequest()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/auth.py", line 51, in __init__
    frappe.local.login_manager = LoginManager()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/auth.py", line 105, in __init__
    if self.login()==False: return
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/auth.py", line 126, in login
    self.authenticate(user=user, pwd=pwd)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/auth.py", line 209, in authenticate
    self.check_if_enabled(user)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/auth.py", line 214, in check_if_enabled
    doc = frappe.get_doc("System Settings")
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 709, in get_doc
    doc = frappe.model.document.get_doc(*args, **kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 69, in get_doc
    return controller(*args, **kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 127, in load_from_db
    if not getattr(self, "_metaclass", False) and self.meta.issingle:
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 37, in get_meta
    meta = Meta(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 84, in __init__
    super(Meta, self).__init__("DocType", doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 89, in load_from_db
    super(Meta, self).load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 149, in load_from_db
    table_fields = self.meta.get_table_fields()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 35, in get_meta
    meta = Meta(meta)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 77, in __init__
    super(Meta, self).__init__(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 113, in __init__
    super(Document, self).__init__(kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 53, in __init__
    self.update(d)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 76, in update
    self.set(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 123, in set
    self.extend(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 160, in extend
    self.append(key, v)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 137, in append
    value = self._init_child(value, key)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 175, in _init_child
    value = get_controller(value["doctype"])(value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 114, in __init__
    self.init_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 240, in init_valid_columns
    for key in self.get_valid_columns():
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 250, in get_valid_columns
    valid = self.meta.get_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 37, in get_meta
    meta = Meta(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 84, in __init__
    super(Meta, self).__init__("DocType", doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 89, in load_from_db
    super(Meta, self).load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 149, in load_from_db
    table_fields = self.meta.get_table_fields()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 35, in get_meta
    meta = Meta(meta)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 77, in __init__
    super(Meta, self).__init__(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 113, in __init__
    super(Document, self).__init__(kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 53, in __init__
    self.update(d)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 76, in update
    self.set(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 123, in set
    self.extend(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 160, in extend
    self.append(key, v)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 137, in append
    value = self._init_child(value, key)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 175, in _init_child
    value = get_controller(value["doctype"])(value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 114, in __init__
    self.init_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 240, in init_valid_columns
    for key in self.get_valid_columns():
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 250, in get_valid_columns
    valid = self.meta.get_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 37, in get_meta
    meta = Meta(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 84, in __init__
    super(Meta, self).__init__("DocType", doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 89, in load_from_db
    super(Meta, self).load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 149, in load_from_db
    table_fields = self.meta.get_table_fields()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 35, in get_meta
    meta = Meta(meta)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 77, in __init__
    super(Meta, self).__init__(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 113, in __init__
    super(Document, self).__init__(kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 53, in __init__
    self.update(d)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 76, in update
    self.set(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 123, in set
    self.extend(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 160, in extend
    self.append(key, v)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 137, in append
    value = self._init_child(value, key)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 175, in _init_child
    value = get_controller(value["doctype"])(value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 114, in __init__
    self.init_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 240, in init_valid_columns
    for key in self.get_valid_columns():
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 250, in get_valid_columns
    valid = self.meta.get_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 37, in get_meta
    meta = Meta(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 84, in __init__
    super(Meta, self).__init__("DocType", doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 89, in load_from_db
    super(Meta, self).load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 149, in load_from_db
    table_fields = self.meta.get_table_fields()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 35, in get_meta
    meta = Meta(meta)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 77, in __init__
    super(Meta, self).__init__(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 113, in __init__
    super(Document, self).__init__(kwargs)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 53, in __init__
    self.update(d)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 76, in update
    self.set(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 123, in set
    self.extend(key, value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 160, in extend
    self.append(key, v)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 137, in append
    value = self._init_child(value, key)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 175, in _init_child
    value = get_controller(value["doctype"])(value)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 114, in __init__
    self.init_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 240, in init_valid_columns
    for key in self.get_valid_columns():
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 250, in get_valid_columns
    valid = self.meta.get_valid_columns()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/base_document.py", line 62, in meta
    self._meta = frappe.get_meta(self.doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/__init__.py", line 734, in get_meta
    return frappe.model.meta.get_meta(doctype, cached=cached)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 37, in get_meta
    meta = Meta(doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 84, in __init__
    super(Meta, self).__init__("DocType", doctype)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 104, in __init__
    self.load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/meta.py", line 89, in load_from_db
    super(Meta, self).load_from_db()
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/model/document.py", line 139, in load_from_db
    d = frappe.db.get_value(self.doctype, self.name, "*", as_dict=1)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/database/database.py", line 441, in get_value
    order_by, cache=cache)
  File "/home/ritesh/Projects-Active/frappe-psql/apps/frappe/frappe/database/database.py", line 490, in get_values
    elif (not ignore) and frappe.db.is_table_missing(e):
  File "/home/ritesh/Projects-Active/frappe-psql/env/lib/python2.7/site-packages/werkzeug/local.py", line 347, in __getattr__
    return getattr(self._get_current_object(), name)
  File "/home/ritesh/Projects-Active/frappe-psql/env/lib/python2.7/site-packages/werkzeug/local.py", line 308, in _get_current_object
    return getattr(self.__local, self.__name__)
RuntimeError: maximum recursion depth exceeded

@surajshetty3416
Copy link
Member

@riteshshrv can you please recheck... I am not getting that error..

Also, If it turns out to be a python 2 specific issue, can you please add commit with a fix.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.06%) to 54.953% when pulling 31df1cb on surajshetty3416:pg-poc into 1a2993e on frappe:develop.

1 similar comment
@coveralls
Copy link

Coverage Status

Coverage increased (+0.06%) to 54.953% when pulling 31df1cb on surajshetty3416:pg-poc into 1a2993e on frappe:develop.

@surajshetty3416 surajshetty3416 changed the title Postgres support in Frappe Postgres support for Frappe Sep 11, 2018
@surajshetty3416 surajshetty3416 merged commit 2e6a202 into frappe:develop Sep 21, 2018
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Apr 28, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants