feat: Virtual DocType

[Thunderbottom]

Summary

Virtual DocType is a feature-extension for DocType which allows developers to create DocTypes with custom data sources and DocType controller. The purpose is to define custom DocTypes in the system without creating a table in the database, while utilizing the frontend, resource APIs, and roles and permissions from the framework.

These Virtual DocTypes function exactly like normal DocTypes in the frontend and are indistinguishable for the end-user, but gives more control to the developer over the DocType's data source. With this, the data source for a Virtual DocType can be anything: an external API, a secondary database, JSON or CSV files, etc. This enables the developers to plug-in database backends other than MariaDB and Postgres, and makes the Frappe Framework even more powerful!

Usage

Creating a Virtual DocType

To create a Virtual DocType, just select the Virtual DocType checkbox while creating the DocType:

Creating a Custom Controller

As an example, the following controller code uses a JSON file as the DocType datasource:

class test_virtual(Document):
	def db_insert(self):
		d = self.get_valid_dict(convert_dates_to_str=True)
		with open("data_file.json", "w+") as read_file:
			json.dump(d, read_file)

	def load_from_db(self):
		with open("data_file.json", "r") as read_file:
			d = json.load(read_file)
			super(Document, self).__init__(d)

	def db_update(self):
		d = self.get_valid_dict(convert_dates_to_str=True)
		with open("data_file.json", "w+") as read_file:
			json.dump(d, read_file)

	def get_list(self, args):
		with open("data_file.json", "r") as read_file:
			return [json.load(read_file)]

To integrate other datasources with the Virtual DocType, you will need to add controller methods defining the database access.

Outcome

• The frontend for Virtual DocTypes remain unchanged:

• All the /api/resource methods defined by the framework are compatible with Virtual DocTypes.

---

Documentation Link: frappe/frappe_docs#71

[szufisher]

more elegant solution than my doctype variant #7946

I am wondering whether it is possible to also combine the doctype variant feature into this one?

the idea is as following

1. add base doctype field DocType, filter only non virtual doctype to be selectable
2. in get_doc and get_list, for virtual doctype with base doctype assigned, call the get_list and get_doc on base doctype, apply the meta on the result.

[Thunderbottom]

I am wondering whether it is possible to also combine the doctype variant feature into this one?

the idea is as following

1. add base doctype field DocType, filter only non virtual doctype to be selectable

2. in get_doc and get_list, for virtual doctype with base doctype assigned, call the get_list and get_doc on base doctype, apply the meta on the result.

You can probably do that with the current implementation of Virtual DocType. Since here you need to define your own DocType controller, you can use any existing DocType as your data source and limit the fields that you want your Virtual DocType to have within the controller itself.

[Thunderbottom]

This PR is ready to be reviewed! @barredterra @surajshetty3416 :)

[szufisher]

Still waiting this to be merged.

[surajshetty3416]

@Thunderbottom you have to enable edit rights for maintainers so that we can push required changes.

[Thunderbottom]

doesn't show the option for me anymore.

[surajshetty3416]

Cannot rely on this, frappe.throw shows an error modal on the client-side which is why UI tests are failing. Why is this required? What was the issue with the previous handling?

[shridarpatil]

frappe.db.exists makes a db call, in case of virtual DocType we wont have a table so it will always return empty.

[shridarpatil]

I was not able to reproduce the error which you mentioned

[shridarpatil]

what do you think is the best way to handle this issue?

[surajshetty3416]

Can we have a exists method for virtual doctype as well? frappe.db.exists can internally use that method to check if virtual document exists.

[shridarpatil]

I don't think its a correct way to handle. Example consider if virtual doctype's backend is an api then we might end up calling same api twice, first call to check if document exists and second call to get the data.

[shridarpatil]

Can we set frappe.local.message_log to false here?

[surajshetty3416]

first call to check if document exists and second call to get the data.

I think that's fine... but it will help in other cases where we are just using frappe.db.exists for different validations.

[shridarpatil]

frappe.db.exists internally uses frappe.db.get_value so I have added support for frappe.db.get_value in virtual doctype,

[surajshetty3416]

LGTM

[surajshetty3416]

@shridarpatil the change might have created some issue for ERPNext patch

Fatal Python error: Cannot recover from stack overflow.
https://github.com/frappe/erpnext/pull/25006/checks?check_run_id=2252328630

Let me know if you are able to figure out and fix it.

[Olawale1]

Hi @Thunderbottom

Thanks a lot for this great addition to Frappe. Could you please guide on the following points (and possibly include in documentation):

1. Where exactly should the controller code be located and is it possible to use Server Scripts for this ?
2. Please add a sample script that could be used if the source is an external database

I think this would be helpful for a lot of other users as well

Thank you kindly

[haolei]

When this feature will be available ? which release will include this feature? I see this feature was merged, but the codes can't be found in any branch

[haolei]

@casesolved-co-uk in v13.1.0, only DocType support virutal_doc feature, but list view, form will not work ( couting in list view, form detail)

[Andriesvn]

For anyone still interested, Applying the following fixes works perfectly.
#13321 d54ecd4 baf4259
Unfortunately still no Report Count so you need to set Disable Count and Disable Sidebarstats on All Virtual Doctype lists