New architecture for record translator

[wszymanski]

Edited by: @wojciechczerniak

Context

WIP

Breaking changes

• modifyRow, modifyCol, unmodifyRow and unmodifyCol will be removed
• manualColumnFreeze will no longer enable manualColumnMove plugin and will work a liitle bit different.
• collapsibleColumns will no longer enable hiddenColumns plugins
• nestedRows will no longer enable trimRows plugin
• filters will no longer enable trimRows plugin
• observeChanges plugin won't be enabled by columnSorting and is from now deprecated
• skipLengthCache hook was removed
• Modifying data by reference and running render() does not work as it used to, because index mappers will not count this line. All operations must be done via Api methods populateFromArray (when cells are not readOnly - more details about that) or setDataAtCell.
• minSpareRows and minRows will ensure that number of visible rows is proper also when the TrimRows plugin is enabled.
• toPhysicalRow / toPhysicalColumn / toVisualRow / toVisualColumn functions won't return non-nullable index for unpresent element anymore.
• Arguments of the afterLoadData has been changed.
• The ManualColumnFreeze plugin works in a different way.
• The RecordTranslator object and plugin's t property which referenced to it have been removed.
• Hooks after..., ie. afterLoadData, afterFilter, afterUnmergeCells are called just before a render call from now.
• Modification of data stored in the table by reference won't be supported anymore.

For further explanations of changes please take a look at #5945.

Progress

State 1

• Create mappers collections where they can be registered
• Register and modify mappers through reference
• All mappers in collection should be updated on remove, insert, loadData, etc.
  • Remove hooks from plugins
• Stop using modifyRow and modifyCol hooks. Use recordTranslator instead.
  • undoRedo plugin
  • dateValidator validator
  • timeValidator validator
• Create physicalCache in IndexMapper class (for O(1) get operations)
• Local Hook to rebuild Cache on changes

Stage 2

• Fix all tests (in stage 1 they were ignored for fast prototyping)
• Should work with minRows, minCols, maxRows, maxCols as before
• Should work with minSpareRows, minSpareCols as before
• Remove old RecordTranslator
• Plugins should no longer use other plugins
  • ManualColumnFreeze should no longer use ManualColumnMove
  • NestedRows should no longer use TrimRows
  • Filters should no longer use TrimRows
• Rewrite all plugins index mapper mixins to register (and use) new IndexMapper
  • TrimRows
  • handsontable/dev-handsontable#167 ColumnSorting and MultiColumnSorting
    • Remove ObserveChanges dependancy
  • Refactor of manual column move #5152 ManualColumnMove
  • Refactor of manual row move #5068 ManualRowMove
  • BindRowsWithHeaders (rewrite strategies as IndexMappers)
  • Remove old IndexMapper mixin
• Rewrite plugins with arrays to use IndexMappers
  • Formulas (nothing to do)
  • Gantt Chart (nothing to do)
  • ColumnSummary (nothing to do)
  • Adjust HiddenColumns to new IndexMapper architecture #6178 HiddenColumns
  • Adjust HiddenRows to new IndexMapper architecture #6177 HiddenRows
  • Adjust ManualRowResize to new IndexMapper architecture. #6164 ManualRowResize
  • Adjust ManualColumnResize to new IndexMapper architecture. #6163 ManualColumnResize
  • Adjust AutoRowSize to new IndexMapper architecture. #6170 AutoRowSize
  • Adjust AutoColumnSize to new IndexMapper architecture. #6169 AutoColumnSize
• Plugins should unregister their index mappers on destroy
• Undo Redo support
• Persistent State support
  • Move ColumnSorting column meta manager to new indexMap
• Batch operations to NOT rebuild Cache on every change

Stage 3a

• CollapsibleColumns should no longer use HiddenColumns plugin #6204 CollapsibleColumns should no longer use HiddenColumns plugin
• Cleanup and Code Review
  • Review all TODO and FIXME comments
  • Remove all magic strings, set them as const
• Write tests for all solved issues (related issues section bellow)
• Create documentation for IndexMappers
  • Update all references to old modifyRow pipeline
  • Update custom plugin tutorial with info how to use IndexMappers
  • Check if all breaking changes are reflected in the docs
• Deprecate observeChanges plugin in docs and in code (warning msg when enabled?)

Stage 3b (WIP by @budnix handsontable/dev-handsontable#130)

• Refactor cellMeta and move it from core.js to data module
  • afterRemoveRow, afterInserRow, afterRemoveCol, afterInsertCol are called before cellMeta is updated #5525, after remove columns get wrong sourceData with formulas enabled #5550, Wrong cell meta removed when deleting rows #6051 CRUD operations doesn't update cellMeta
  • add setColMeta, getColMeta methods #4902, add setRowMeta, getRowMeta methods #4901 Row / Column meta data based on indexMapper
  • Cell meta has more dimensions than indexMapper, has to have own mechanism for that

Releated issues

Fixed:

1. • Use of column sorting causes unexpected rendering when the underlying data structure is modified #2685 columnSorting should work without observeChanges
2. • manualMoveRow don't work after loadData #3568 manualMoveRow don't work after loadData
3. • Inserting a row after sorting and loadData breaks index translation and results in additional blank rows #3809 CRUD operations should work with columnSorting and loadData
4. • Blank Rows in Middle of Grid [PRO] #3937 Blank Rows in Middle of Grid [PRO]
5. • сolumnSummary does not work with columnSorting and context menu #3924 сolumnSummary does not work with columnSorting and context menu
6. • Using loadData on an object data doesn't work #4204 Using loadData on an object data doesn't work
7. • Hidden columns become visible when we run updateSettings #4121 hiddenColumns settings are reseted after updateSettings
8. • Filtering possibly uses the wrong indexes #4442 Filtering is using wrong indexes
9. • [0.28.4+] Moved column displays incorrect values #4470 Wrong value after column move
10. • afterRowMove hook returns improper target parameter #4501 afterRowMove hook returns improper target parameter afterRowMove hook returns improper target parameter #4501
11. • manualColumnFreeze enables manualColumnMove #4553 manualColumnFreeze should work without manualColumnMove
12. • Plugins using arrayMapper may don't work properly #4567 recordTranslator shouldn't return null - need more test
13. • [0.27.0+] Data not render when use manualRowMove and minSpareRows #4576 [0.27.0+] Data not render when use manualRowMove and minSpareRows
14. • [0.28.0+] columnSorting: true breaks an ability to freeze columns properly. #4601 manualColumnFreeze should work with columnSorting
15. • colWidth will be wrong when inserting or removing column after resized, rowHeight too. #4666 Manual column width doesn't follow column if we insert / remove column
16. • [0.28.1+] Row moving doesn't work with column sorting when PersistentState is enabled #4713 Row moving doesn't work with column sorting when PersistentState is enabled
17. • Column reordering not working properly with an empty data source or not valid objects in columns. #4926 manualColumnMove doesn't work when data object is empty
18. • collapseAll method for collapsibleColumns doesn't work with sorting enabled (observeChanges) #4999 collapsibleColumns doesn't work with columnSorting
19. • Global index mapper #5112 Global index mapper (EPIC, nothing to do here)
20. • Common RowsMappers for plugins using such the structure #5065 Common rowsMapper for columnSorting manualRowMove and trimRows - nothing to test
21. • Refactor of manual row move #5068 Refactor manualRowMove - nothing to test
22. • Filter causes Minsparerows to have problems #5707 Filter causes Minsparerows to have problems
23. • problem with afterColumnMove #5173 problem with afterColumnMove
24. • Filtering doesn't work after moving rows #5178 manualRowMove should work with filters
25. • Handsontable colHeaders gets NaN on loadData #5369 Handsontable colHeaders gets NaN on loadData
26. • [5.0.2+] htInvalid shifts when row are added/removed #5674 htInvalid should be updated on CRUD operations
27. • skipColumnOnPaste is broken when columnSorting and hiddenColumns are enabled #5824 skipColumnOnPaste is broken with columnSorting enabled
28. • Trim rows plugin doesn't cooperates properly with manualRowMove plugin #5860 trimRows should work with moveRow
29. • minSpareRows does not appear if we set at least one row as trimmed #5862 trimRows does not work with minSpareRows
30. • Filters doesn't work with nestedRows #5889 nestedRows should work with filters
31. • Filtering and sorting don't work together #5890 filters should work with columnSorting
32. • Filters and trimRows plugins can't work simultaneously #5915 trimRows should work with filters
33. • manualColumnMove does not work for small object data source #5931 manualColumnMove doesn't work when columns count is different from array of objects properties count
34. • TrimRows plugin breaks expected startRows behaviour #5953 trimRows should work with startRows
35. • Sorting breaks adding new rows #5956 columnSorting is not working correctly with loadData
36. • afterColumnMove/beforeColumnMove doesn't tell you where the columns move to #6005 afterColumnMove/beforeColumnMove doesn't tell you where the columns move to
37. • Filters and minSpareRows problems at the same time #6278 Filters and minSpareRows problems at the same time - additional error fixed
38. • Duplicated rows while using manualRowMove with improper index #6088 Duplicated rows while using manualRowMove with improper index
39. • Visual indexes are wrong when using trimRows and columnSorting #6310 Visual indexes are wrong when using trimRows and columnSorting
40. • updateSettings in afterColumnMove error #4480 updateSettings in afterColumnMove
41. • loadData() breaks column filters #5244 loadData() breaks column filters
42. • nestedRows does not allow to keep rowHeaders after collapsing #5874 nestedRows does not allow to keep rowHeaders after collapsing
43. • Wrong position of buttons in nestedRows after use detach from parent #5900 Wrong position of buttons in nestedRows after use detach from parent
44. • ColumnSummary: Extra row + unchangeable values #5794 ColumnSummary: Extra row + unchangeable values
45. • [7.1.0] Error when we move MULTIPLE rows in a nestedRows table #6066 move multiple rows with nestedRows
46. • [7.1.1] Error when columnSummary plugin tries to create a row #6300 doesn't work with columnSummary when plugin tries to create a row
47. • Add loadData support for Nested Rows. #6339 Add loadData support for Nested Rows.
48. • nested rows doesn't work with loadData #3914 nested rows provokes error when removing and adding rows
49. • Once we enable columnSorting we cannot run the merge() method #6086 Once we enable columnSorting we cannot run the merge() method
50. • After removing column rowHeaders stay improperly #6412 After removing column rowHeaders stay improperly

Regression:

None.

Currently tested:

• Performance and CPU wastage issue due to some handsontable properties. #6058 Performance and CPU wastage issue due to some handsontable properties.

Cell Meta

Moved to #6274

Not related to 5751

1. Sorting passing logical index through afterChange event #1501 Sorting passing logical index through afterChange event
2. #3941 Auto column size plugin doesn't work properly after setting maxRows (by updateSettings)
3. Improper selection after insert column/row when mergeCells is enabled. #4897 Wrong selection after inserting a row / column with mergeCells plugin enabled - bug reproduced
4. Wrong 3rd parameter return for the beforeRemoveRow hook #5100 Wrong 3rd parameter return for the beforeRemoveRow hook - documentation changed
5. Changing data while in edit results in edit to wrong data #5270 Changing data while in edit results in edit to wrong data - additional error
6. Overloaded pluginHookBucket because of updateSettings or loadData #5727 We shouldn't use interval to calculate cache length there was a disscusion
7. Filtering results in blank rows #5208 Filtering results in blank rows - hot.getPlugin("Filters").filter() results
8. ManualColumnMove dragging breaks after updateSettings() #6004 ManualColumnMove dragging breaks after updateSettings()
9. Rendering issue on column udpate with updateSettings #3770 Rendering issue on column udpate
10. When we enable filters we aren't able to add new rows while pasting data #5648 filters don't work with copyPaste when new row has to be created bug reproduced
11. Sorting after filtering with minSpareRows set #5983 Sorting after filtering with minSpareRows set
12. [7.1.0.+] We cannot move nested row to become the last one in the table #6067 We cannot move nested row to become the last one in the table
13. Headers doesn’t export with exportToFile in the specific case  #4176 countSourceCols omits empty cells
14. after remove columns get wrong sourceData with formulas enabled #5550 With enabled formulas the sourceData is not updated after remove column
15. Issue with setDataAtRowProp() - bad column index translation #4118 Issue with setDataAtRowProp() - bad column index translation

Found and repaired immediately

1. • [8.0.0-rc1] Visual width is improper after moving column #6292 [8.0.0-rc1] Visual width is improper after moving column
2. • [8.0.0-rc1] Newly inserted child row overwrites existing rows #6320 [8.0.0-rc1] Newly inserted child row overwrites existing rows
3. • [8.0.0-rc1] Can't collapsed many parent of nestedRows. #6316 [8.0.0-rc1] Can't collapsed many parent of nestedRows.
4. • [8.0.0-rc1] Cells in Handsontable Editor no longer contain 'listbox' class. #6322 [8.0.0-rc1] Cells in Handsontable Editor no longer contain 'listbox' class.
5. • [8.0.0-rc1] Column headers stay improperly after removing rows #6410 [8.0.0-rc1] Column headers stay improperly after removing rows

After HiddenColumns

Moved to #6178

Checklist

• Affected plugins:

  • trimRows
  • nestedRows
  • manualColumnFreeze / manualRowFreeze
  • filters
  • manualRowMove / manualColumnMove
  • hiddenRows / hiddenColumns
  • manualRowResize / manualColumnResize
  • autoRowResize / autoColumnResize
  • columnSorting / multipleColumnSorting
  • customBorders
  • collapsibleColumns
  • PersistantState
  • observeChanges in wrappers

• Core functionalities:

  • updateSettings
    • What if I change columns property
    • What if columns is a function
    • Does it work with minSpareRows and minSpareCols
    • Does it work with minRows, minCols, maxRows, maxCols
  • copy, cut and paste
  • Cell validation
  • CRUD operations, insertCol, removeCol, insertRow, removeRow
    • Performance if we inser/remove 10 000 cols / rows vs Handsontable 7.1
    • Move rows between fixed top, fixed bottom and main overlays
    • Move columns between fixed left and main overlays

[aaronbeall]

Just want to say: I've been watching progress on this issue and the work is really impressive. The number of fixed issues are astounding. It's rare (in my experience) to see such a foundational refactoring come out alive. :)

[wojciechczerniak]

Thanks @aaronbeall ! We took our experience with the product from recent years, feedback from you and other developers, and decided to put it into good use. More is coming after this refactor. We're not done yet 😄