Agent Backend for Royale Form Builder

[JoelProminic]

We originally planned to have an agent backend for #704. We would generate agents for each Form, build them with NSFODP, and then build and deploy them in a Vagrant Domino instance (or configured Domino server?).

To support this, we need:

• Templates for the different agents. I can use the basic design from FlexAgentBase for this. I'll plan to leave a hook for authentication, but it should allow all access by default
• Generate the agents to an On Disk Project
• Create a dedicated Vagrant script for Domino and NSFODP. I have code to create the Domino server in other scripts
• Communication between Moonshine and the Vagrant VM through the REST interface.

UPDATE: Branch information:
Branch: features/issue_978_CRUD_agents_integration
Build: https://moonshine-ide.com/downloads/releases/dev/macos/branches/features/issue_978_CRUD_agents_integration
Test Project: https://github.com/Moonshine-IDE/FormBuilder-Example-AuthorBookFan

[JoelProminic]

I started on the agent templates in https://github.com/Moonshine-IDE/Moonshine-Domino-CRUD

I created issues for individual features on that project. I'll update this issue when I have an initial working issue ready.

[JoelProminic]

There was some confusion about the role of the agents today. Here are some notes from the discussion.

The overall goal is to have a working Royale interface for a Domino database. To implement this, we need these components

1. Generate the Royale UI (Done)
2. Generate a Royale layer to call and parse the agents
3. Generate the Domino agents (In progress)
4. Generate the Domino database (Done)
5. Deploy the agents to the database
6. Deploy the agents to a server

Here is a diagram of how the components work together:

[JoelProminic]

I created an early version of the template for the Java agents: https://moonshine-ide.com/downloads/Moonshine-Domino-CRUD/dev/template.zip

This TemplateREADME.md, also included in the zip, explains the structure of the template, and how to use the template.

I wrote up an example of the template generation logic in Groovy. Hopefully this will be useful at least as pseudocode to clarify the algorithm.

For now, add a new action in the project menu called "Generate Java Agents". This should

1. Download template.zip from the above URL. See more on this below
2. Unzip template.zip to a temporary directory
3. Run the algorithm against this template to generate the Java project.
   • Prompt the user to select the parent directory
   • For the project name, use %OriginalProjectName%_JavaAgents
4. Open the resulting project in Moonshine
5. Cleanup the template

My goal for downloading the template is:

• No need to commit or maintain generated template in Moonshine-IDE.
• Option to update template without updating Moonshine-IDE. This will be especially useful early on when I am making more template updates.
• Updating the template is simple if the algorithm, categories, or insertion parameters don't need to be changed. Some options:
  • Replace zip on the server
  • Replace URL in a configuration file on the server
  • Update the URL in the code - this will be the most flexible option for versioning

The template is very small currently (32 KB), so the download shouldn't be a problem.

The template may eventually be read from an official release on https://github.com/Moonshine-IDE/Moonshine-Domino-CRUD, but the project is private for now.

Let me know if you have any questions or concerns for this. For now, this update is limited to generating the Java agents. There are may other updates pending:

• Generate a Domino server with Vagrant
• Deploy the generated NSFODP database to the Domino server
• Deploy the generated agents to the Domino server (probably DXL Importer: DXL Importer Project for Domino Agent Support #668, Minimal DXL Importer Project Example #935)
• Royale code to interact with the CRUD agents.
• More agent updates to support more complicated cases in Form Builder (and eventually Domino Visual Editor)

I'd like to let @rat-moonshine start working on the Royale code soon. My plan for this is to supply:

• Create an example FormBuilder project
• Manually deploy agents to a test Domino server. These agents would be generated with the above logic. I would include some utility agents including an agent to clear the documents
• Document the agent API. Currently this is XML, but I'm thinking I will switch to JSON before I deliver the API. Writing test cases for Moonshine-Domino-CRUD has reminded me how much of a pain it is to retrieve values from XML, and how messy the format is for the current Document XML.

[JoelProminic]

Here is an example project generated with the Groovy class:

Project: 2022_06_10__CRUDExample.zip

From this configuration:

		Form form = new Form(name:'Author', view:'Authors')
		project.forms << form
		
		form.fields << new Field(name:'AuthorID',
			                        label:'ID',
			                        description:'Unique identifier for lookups',
			                        type:'Text',
			                        multivalue:false,
			                        editable:'Editable',
			                        required:false,
			                        includeInView:true,
			                        sorted:"Ascending"
		)
		
		form.fields << new Field(name:'Name',
			                        label:'Name',
			                        description:'Full name',
			                        type:'Text',
			                        multivalue:false,
			                        editable:'Editable',
			                        required:false,
			                        includeInView:true,
			                        sorted:'No sorting'
		)

UPDATE: I also added some debugging lines in GenerateTemplate.groovy so you could see the logic flow:

Processing directory 'build/template/project'
Processing template file 'build/template/project/build.gradle.template'
Processing directory 'build/template/project/lib'
Processing plain file:  'build/template/project/lib/Moonshine-Domino-CRUD.jar'
Processing template file 'build/template/project/%project%.javaproj.template'
Processing directory 'build/template/project/src'
Processing directory 'build/template/project/src/main'
Processing directory 'build/template/project/src/main/java'
Processing %eachform% directory:  'build/template/project/src/main/java/%eachform%Agents'
Generating the 'all/' insertion parameters for form 'Author'
Generating the parameter 'all/FieldList' for form 'Author'
Generating the 'key/' insertion parameters for form 'Author'
Generating the parameter 'key/KeyList' for form 'Author'
Generating the 'editable/' insertion parameters for form 'Author'
Generating the parameter 'editable/FieldList' for form 'Author'
Generating the 'required/' insertion parameters for form 'Author'
Processing directory 'build/template/project/src/main/java/%eachform%Agents'
Processing template file 'build/template/project/src/main/java/%eachform%Agents/%form%Read.java.template'
Processing template file 'build/template/project/src/main/java/%eachform%Agents/%form%Delete.java.template'
Processing template file 'build/template/project/src/main/java/%eachform%Agents/%form%Create.java.template'
Processing template file 'build/template/project/src/main/java/%eachform%Agents/%form%Update.java.template'

[JoelProminic]

I tested the updates today, and the generated agents compile and work properly. I tested

• Create a new project with a single form
• Create a new project with multiple forms
• Compile the generated project (worked after I fixed notesExecutablePath - see below)
• Copy the agents to a test database and confirm that the operations work

As discussed in the morning meeting, these features aren't working as expected:

• When running Project > Generate Java Agents, I have to select a blank directory for the new project. Instead, this should select the parent directory, and create a project called %OriginalProjectName%_JavaAgents
• Open the generated project in Moonshine automatically
• Currently, %NotesExecutablePath% in build.gradle is populated as /Applications/HCL Notes.app. Instead, this should be Applications/HCL Notes.app/Contents/MacOS/. This should behave the same way as $NotesExecutablePath in Java Domino Project

[rat-moonshine]

Updates:

• %OriginalProjectName%_JavaAgents named directory creation by selecting parent directory
• Project name fix
• Notes paths fix on macOS
• Console message and codes cleanup

[JoelProminic]

While testing during the meeting, I got this error:

: Error #1006
: TypeError: Error #1006
: 	at actionScripts.plugins.ondiskproj.crud.exporter::CRUDJavaAgentsModuleExporter/copyModuleTemplates()
: 	at Function/actionScripts.plugins.ondiskproj.crud.exporter:CRUDJavaAgentsModuleExporter/protected:parseModules/actionScripts.plugins.ondiskproj.crud.exporter:onFilesParseCompletes()
: 	at Function/actionScripts.utils:UtilsCore/parseFilesList/actionScripts.utils:onFilesListParseCompleted()
: 	at flash.events::EventDispatcher/dispatchEvent()
: 	at Function/actionScripts.utils:FileSystemParser/protected:listOfProcessEnded/actionScripts.utils:onReadCompletes()
: 	at Function/actionScripts.utils:FileUtils/readFromFileAsync/actionScripts.utils:onFileReadCompletes()
:
: Click here to Report a Bug

Here is the project:
2022_06_15__TestFormBuilder.zip

However, I tested this against the same project earlier today, but I didn't see the error.

[JoelProminic]

The Moonshine-Domino-API is getting close.

The next step after this will be to generate an example FormBuilder project and manually import the agents so that @rat-moonshine can start testing the Royale implementation.

The initial example project should be simple, but I'll eventually want to make the project more complex so that I can different special cases like:

• All different Field types
• Multiple keys
• Computed fields
• (Computed key](https://github.com/Moonshine-IDE/Moonshine-Domino-CRUD/issues/13)

[JoelProminic]

@rat-moonshine, is there anything still pending on the agent generation side? If not, please merge it for tomorrow so that we can work from the master build only in the meeting/demo tomorrow.

One remaining issue I can think of is finding a "final" location for the template file, but this would be easy to change after the merge (as long as we remember).

There is still a lot to do to integrate the agents in Royale. We can continue this in a new branch and/or new issue.

[rat-moonshine]

This issue is now merged into master.

[rat-moonshine]

I encountered with different problems today while testing the backend agents in generated Royale project. This makes me thinking whether I'm using the correct URL for the purpose?

Currently, I'm using test URLs from @JoelProminic from domino-49.prominic.net, and I see following problems.

1. I encountered with CORS issue while running the application on localhost:3000. Although I'm using service.addBead(new CORSCredentialsBead(true)); code while submitting the HTTPService as we do in MyAccount, I'm not sure if I need to make any other adjustment in someplace else.
   
2. Testing addition of a new entry on browser (e.g. https://domino-49.prominic.net/A55606/Joel/Test-Moonshine-Domino-CRUD.nsf/AuthorCreate?OpenAgent&AuthorID=AUTHOR-012&NameFirst=Jane&NameLast=Doe&Age=25), I was expecting it'll return an universal ID but I don't seen any:
   
3. The listing agent doesn't returns universal IDs either:
   

Maybe the new logic isn't accessible because it is not released yet (?)

[JoelProminic]

In order to resolve the CRUD errors, I had to move the agents to rest-swf.prominic.net. The base URL is now: https://rest-swf.prominic.net/A55D78/Test-Moonshine-Domino-CRUD.nsf

In addition, we'll need to run the test application from a local server, with a corresponding hosts file entry for localdev.prominic.net. We are doing something similar for MyAccount. We couldn't use the default server started by Moonshine because the CORS entry on rest-swf specifically requires https for localdev.prominic.net.

[JoelProminic]

I have released v0.3 of Moonshine-Domino-CRUD, which has changes for the keys.

Instead of using the sorted columns from FormBuilder for the lookup keys, the agents now expect DominoUniversalID as the key. This is the ID generated by the Domino backend.o

You can retrieve DominoUniversalID from the document object in the Create agent response, or from the Read agent.

Note that the template still uses the key category for %eachform%, so we still need to generate the key list. However, it is a low priority, since this key shouldn't be used by the default generated interface. The remaining uses are:

• The Read agent supports an option "filtered" lookup by the keys. The keys are not necessarily required to be unique in this case, but they do require a view (which is already generated).
• I included comments in the Create, Update, and Delete agents to show how to enable the custom keys if desired.

[rat-moonshine]

I should had add this earlier, @JoelProminic you can download the branch build for macOS to test the updates. Today I able to integrate only listing, addition and deletion agents.

Please note, in OnDisk project's Settings > Domino > Base URL provide the URL value till .nsf path, i.e. https://rest-swf.prominic.net/.../Some.nsf. I shall be joining soon.

[JoelProminic]

I moved the AuthorBookFan project to GitHub for more convenient updates: https://github.com/Moonshine-IDE/FormBuilder-Example-AuthorBookFan

Updates:

• Disabled Show in View for Date, Multivalue, and RichText fields. These can be added back if desired.
• Fixed Author.Updates to be a multivalue DateTime field.

Once we have this example working, we can add additional features like:

• Computed fields - these should be read from the agents, but not updated.
• Multi-Key Lookups - this is only relevant for the Read agent, and not used by the base design. We may revisit this for ViewBuilder.

[rat-moonshine]

@JoelProminic I wonder how you compiles the generated Royale project from this application in Moonshine-IDE (?) Using hyphen in class-names seems not identified as valid identifier:

: /Users/devsena/FormBuilder-Example-AuthorBookFanRoyaleApplication/src/FormBuilder-Example-AuthorBookFanRoyaleApplication.mxml(2): col: 2 Error: The component name 'FormBuilder-Example-AuthorBookFanRoyaleApplication' is not a valid ActionScript identifier.
: <j:Application xmlns:fx="http://ns.adobe.com/mxml/2009"
:  ^

[rat-moonshine]

Updates:

When the Book module now works correctly in CRUD, there's something with the Author agents. It seems like I'm sending correct values but the add and update agents are still returning errors, I wonder if this related with date-fields involvement (?)

Author Add Request:

Author Add Response:

Author Update Request:

Author Update Response:

The branch build is now ready for a testing.

[rat-moonshine]

Updates:

• Array submission fix to agent
• Back button icon changed
• Waiting in form view until server returns
• Column renderers for multivalue-datefield/string/number (currently, showing as simple string separated by commas)
• But fix where a successful update following a failure update did not updates the record in grid immediately

[JoelProminic]

Array submission is working when values are provided for Text, Number, and DateTime multivalue fields. However, if no value are provided, the Royale application submits "null" which is an invalid value. Instead, it should submit an empty array: []

Similarly, I see that if a DateTime value is not set, the parameter is passed as "null". I was originally planning to have you not set the parameter in this case, but this does not work if we need some way to remove the value for a DateTime field on the Domino side. This is not something I do a lot with my agents, so I need to do some more planning for this.

The Back icon looks fine now.

I see that the application is staying in the form if there is an error, but it is displaying errorMessage in a snackbar message. Instead, I want to display the errorMessage in red just above the Submit and Cancel buttons.

The "Request submitted successfully" snackbar message should not trigger a snackbar message on successful call to the Read agent (page load or Refresh button). We can keep this for now if you need it for debugging, but I originally thought this was a bug with the status of the application.

The column values for DateTime and Multivalue fields look mostly good, but I see that it still shows [object Object] when the DateOfBirth field does not have a value. This should be displayed as a blank cell instead. We may revisit this as part of the DateTime discussion above.

[JoelProminic]

I see that I forgot to writeup the validationErrors property. The JSON looks like this:

{
    "errorMessage":"At least one field is invalid.",
     "validationErrors":[
        {"fieldName":"NameFirst","message":"NameFirst is required."},
        {"fieldName":"NameLast","message":"NameLast is required."}
    ],
    "state":"authenticated",
    "username":"Anonymous"
}

Property	Description
fieldName	the name of the corresponding field. null indicates no corresponding field. Multiple ValidationErrors may be returned for the same field
message	The validation message to display

For now, I would like to see these errors displayed below the errorMessage at the bottom of the form. In the long term, we could display them next to the corresponding field.

The default validation logic only validates the type of fields. To support testing validationErrors, I added some code to trigger errors if AuthorID, NameFirst, or NameLast are blank.

@Override
protected String validate() {
	String response = super.validate();
	boolean hasErrors = !DominoUtils.isValueEmpty(response);
	
	if (DominoUtils.isValueEmpty(getParameter("AuthorID"))) {
		addValidationMessage("AuthorID", "AuthorID is required.");
		hasErrors = true;
	}
	if (DominoUtils.isValueEmpty(getParameter("NameFirst"))) {
		addValidationMessage("NameFirst", "NameFirst is required.");
		hasErrors = true;
	}
	if (DominoUtils.isValueEmpty(getParameter("NameLast"))) {
		addValidationMessage("NameLast", "NameLast is required.");
		hasErrors = true;
	}
	
	if (hasErrors) {
		return "At least one field is invalid.";
	}
	else {
		return null;
	}
}

[rat-moonshine]

Updates:

• Array without an item now submits as []
• Snackbar message during load-list now removes
• errorMessage and validationErrors are now showing at bottom of the form
• Somehow managed to add/remove a busy-indicator with disabled view

On column value showing as [object Object], this probably the same problem that I reported few days ago to @piotrzarzycki21 when a property value is null or the property doesn't exists (DataGrid printing columns as [object Object] from 08/04/2022). Once this resolved at SDK side then we'll probably need not to worry about it.

[JoelProminic]

I tested the updates

• Array without an item now submits as [] - Even with the updates after today's meeting, I'm seeing that the empty multivalue fields are passed as and empty string instead of []. This is producing the same result for the create agent, but it is a bit ambigious, so I'd prefer to use []
• Snackbar message during load-list now removes - Confirmed
• errorMessage and validationErrors are now showing at bottom of the form - Confirmed. This looks good for now (see example below).
• Somehow managed to add/remove a busy-indicator with disabled view - I haven't seen an issue trigger for this yet.
  

[rat-moonshine]

Even with the updates after today's meeting, I'm seeing that the empty multivalue fields are passed as and empty string instead of []

I re-adjusted the generation, following are few test cases:

Author-Create

Author-Update

Book-Create (Note: I see agent returns success event even if I don't submit with any value)

Book-Update

[JoelProminic]

I confirmed that the empty multivalue parameters were fixed. This is ready to be merged. We will start on similar logic for Domino Visual Editor.

[rat-moonshine]

This is now merged into master.