[Metadata Management] Updating ProductTemplates is not correct

[maxceem]

ProductTemplates have a root field called questions instead of sections. But when we display them in the Metadata Editor we show their JSON with questions replaced with sections:

This most likely happens because of these lines https://github.com/appirio-tech/connect-app/blob/dev/src/routes/metadata/components/MetaDataPanel.jsx#L595-L597

If we edit something in such ProductTemplate JSON and save it, it would be saved with sections root element instead of questions.

For example, if I do it locally for a demo template. When I load it the template it has questions as a root element:

But when I save it, it is saved with sections as a root element:

As a result, on the server we have a product template with two root elements questions and sections.

As client side we use only questions for productTemplates, the change would make effect.

[maxceem]

Fixed in 6f7591e

fyi @vikasrohit

[vikasrohit]

Thanks @maxceem for catching the issue and fixing it super fast. However, for better understanding, I would like to reconfirm the issue. If I understood correctly, saving any existing template causes the actual template details (i.e. sections and subsections) saved as new sections field while questions still contains old values and our front end reads from questions field. Also, when we edit a product template in Metadata editor, we prefer (before your changes) questions over sections so essentially any thing that a user changed in a product template, won't persist as we show first available field of questions and sections in the JSON editor.

[maxceem]

Thanks @maxceem for catching the issue and fixing it super fast. However, for better understanding, I would like to reconfirm the issue. If I understood correctly, saving any existing template causes the actual template details (i.e. sections and subsections) saved as new sections field while questions still contains old values and our front end reads from questions field.

Right.

Also, when we edit a product template in Metadata editor, we prefer (before your changes) questions over sections so essentially any thing that a user changed in a product template, won't persist as we show first available field of questions and sections in the JSON editor.

Right, as per logic that I see, a user would always make changes to the initial version of productTemplate which was put into questions, but the update would be always saved to sections which would be always ignored and never used.

[vikasrohit]

Understood. However, as per your fix, we are not always reading questions from the request and rekeying it as sections just for editor screen i.e. there would be no sections key in actual JSON stored in the database, which is kind of good if we don't to break things. However, my intention is to migrate all product templates to use sections instead of questions as key in the template field. I was thinking of metadata editor in a way that every time we save a product template, it saves template content in sections key and questions key does not exist any more after that. While on the front end we still read both sections and questions, preferring the sections over questions. If we don't have the front end part already in place, I guess we have to live with the current changes you made (as it would at least keep admins under impression that both product template and project templates have sections key)..

[maxceem]

However, as per your fix, we are not always reading questions from the request and rekeying it as sections just for editor screen i.e. there would be no sections key in actual JSON stored in the database, which is kind of good if we don't to break things.

The code which rekeys questions to sections is only for the wizard logic which is not visible to users:

So this is just a technical fix which users are not aware of.

In the JSON editor, we show the actual JSON data with questions:

However, my intention is to migrate all product templates to use sections instead of questions as key in the template field.

Yeah, I was also thinking about that to make things easier.

While on the front end we still read both sections and questions, preferring the sections over questions. If we don't have the front end part already in place, I guess we have to live with the current changes you made (as it would at least keep admins under impression that both product template and project templates have sections key)..

Yeah, the thing is that in the frontend we always read only from questions for productTemplates. So we have two ways to go:

1. Just migrate at one moment all the productTemplates to sections by updateding clien-side code and updating templates in the DB.
2. As you suggested we can adjust the client side code to use sections with the fallback to questions. And after fix Metadata editor, so it would replace questions with sections when we open any template as it was done before and when we save we should save only sections. As backend code merge data instead of replacing we should set question to null or undefined to force removing the old key from the backend to avoid confusion in the future.

[vikasrohit]

Thanks @maxceem for the update. Which way do you prefer? I am inclined towards 2 as it is not very straightforward to update all product template's JSON without writing a small script for doing that. If we can do that using migration sqls, I think we can go for 1 as well.

[maxceem]

@vikasrohit I prefer the 1st way. As we already have quite many logic branches depend on project/templates versions and will have more when we introduce forms, priceConfigs and planConfigs client-side. So testing all the possible cases getting harder.

I'm not sure if we can do that using only SQL, though maybe we can. Even if need to write a small Node script I would still prefer to migrate to it into a single moment and update code to support only one way. In such case we would see if we migrated properly or have any issues immediately and could address them immediately. Supporting both ways could potentially lead to having some "silent" issues like the current issue.

[vikasrohit]

Okay, lets go with 1st and have the script ready for updating all templates. If we can think of having it documented in some way for the records, e.g. having a folder for such one way migrations, would be good for future references, however, that is not critical for now.