Add bitemporal data structure visualizer #94

wata727 · 2022-06-23T10:44:37Z

Although the Bi-temporal data model is a good data structure, it has a difficult problem to intuitively understand the data structure that has two times, valid time and transaction time.

This PR introduces a visualizer that plots multiple histories on a two-dimensional figure to help you understand the data structure. This visualizer draws the following history as follows:

valid_from	valid_to	transaction_from	transaction_to
2020-07-31 01:43:17.119	9999-12-31 09:00:00.000	2020-07-31 01:43:17.120	2020-07-31 01:43:18.243
2020-07-31 01:43:17.119	2020-07-31 01:43:18.243	2020-07-31 01:43:18.243	9999-12-31 09:00:00.000
2020-07-31 01:43:18.243	9999-12-31 09:00:00.000	2020-07-31 01:43:18.243	2020-07-31 01:43:18.324
2020-07-31 01:43:18.243	2020-07-31 01:43:18.324	2020-07-31 01:43:18.324	9999-12-31 09:00:00.000
2020-07-31 01:43:18.324	9999-12-31 09:00:00.000	2020-07-31 01:43:18.324	9999-12-31 09:00:00.000

[98] pry(main)> puts ActiveRecord::Bitemporal::Visualizer.visualize(employee)

                        | 2020-07-31 01:43:17.119
                        |                 | 2020-07-31 01:43:18.243
                        |                 | | 2020-07-31 01:43:18.324
                        |                 | |                   | 9999-12-31 09:00:00.000
2020-07-31 01:43:17.120 +---------------------------------------+
                        |                                       |
                        |                                       |
                        |                                       |
2020-07-31 01:43:18.243 +-----------------+---------------------+
                        |                 |                     |
2020-07-31 01:43:18.324 |                 +-+-------------------+
                        |                 | |*******************|
                        |                 | |*******************|
                        |                 | |*******************|
9999-12-31 09:00:00.000 +-----------------+-+-------------------+

The area filled with * is the duration of the instance passed to visualize. This makes it easier to understand where the history in the overall history.

Apart from visualize, visualize_records is also provided as a lower-level interface. This method allows you to pass histories and another history to highlight. The following is example:

ActiveRecord::Bitemporal::Visualizer.visualize_records(Employy.ignore_bitemporal_datetime.where("'2019-01-12' < valid_from"), [employee])

In the above example, unlike visualize, you can only draw histories after 2019-01-12, not all history. Alternatively, you can draw the temporal relationships of different associations.

ActiveRecord::Bitemporal::Visualizer.visualize_records(Employee.ignore_bitemporal_datetime, [department])

osyo-manga · 2022-06-23T14:52:54Z

lib/activerecord-bitemporal/visualizer.rb

+    module_function
+
+    def visualize(record, height: 10, width: 40, highlight: true)
+      histories = record.class.ignore_valid_datetime.within_deleted.bitemporal_for(record)


You can use ignore_bitemporal_datetime instead of ignore_valid_datetime.within_deleted.

Suggested change

histories = record.class.ignore_valid_datetime.within_deleted.bitemporal_for(record)

histories = record.class.ignore_bitemporal_datetime.bitemporal_for(record)

osyo-manga · 2022-06-23T14:57:16Z

lib/activerecord-bitemporal/visualizer.rb

+      valid_times = (histories.map(&:valid_from) + histories.map(&:valid_to)).sort.uniq
+      transaction_times = (histories.map(&:transaction_from) + histories.map(&:transaction_to)).sort.uniq


Can you change it to use pluck(:valid_from)?
because #pluck is better performance

If the histories are already loaded, ActiveRecord::Calculations#pluck is just the same as map(&:valid_from).
https://github.com/rails/rails/blob/v7.0.3/activerecord/lib/active_record/relation/calculations.rb#L194
https://github.com/rails/rails/blob/v7.0.3/activesupport/lib/active_support/core_ext/enumerable.rb#L201

I believe that map(&:valid_from) is better in performance because it is much more efficient in this case to iterate the array than to execute SQL.

Okey, Thanks :)

osyo-manga · 2022-07-04T04:45:58Z

Thanks :)
It's great!

I have a few questions about a few things that have been bothering me.

What do * and # mean?
I think it would be easier to see if I add | as follows, is it possible to implement it?

                        | 2020-07-31 01:43:17.119
                        |                 | 2020-07-31 01:43:18.243
                        |                 | | 2020-07-31 01:43:18.324
                        |                 | |                   | 9999-12-31 09:00:00.000
2020-07-31 01:43:17.120 |---------------------------------------|
                        |                                       |
                        |                                       |
                        |                                       |
2020-07-31 01:43:18.243 |-----------------|---------------------|
                        |                 |                     |
2020-07-31 01:43:18.324 |                 |-|-------------------|
                        |                 | |*******************|
                        |                 | |*******************|
                        |                 | |*******************|
9999-12-31 09:00:00.000 |-----------------|-|-------------------|

wata727 · 2022-07-04T05:09:48Z

* indicates where in the entire history the passed history is and emphasizes it. In the above example, it represents a history where valid time is 2020-07-31 01:43:18.324 ~ 9999-12-31 09:00:00.000, transaction time is 2020-07-31 01:43:18.324 ~ 9999-12-31 09:00:00.000 was passed to the argument of visualize. This helps you understand where a history is in the overall history.

# is a zero-length history representation such as valid_from = valid_to or transaction_from = transaction_to. From my experience, zero-length history can cause a lot of problems, so I find it useful to visualize this.

I think it would be easier to see if I add | as follows, is it possible to implement it?

Yes, but the reason I didn't do that was that the header part might look like part of a history. If you don't have this concern, we may implement this.

osyo-manga · 2022-07-04T05:29:23Z

Thanks for reply.

Yes, but the reason I didn't do that was that the header part might look like part of a history. If you don't have this concern, we may implement this.

I don't see a problem.

wata727 · 2022-07-04T11:16:18Z

Fixed by 95a19ef

wata727 · 2022-07-06T10:00:20Z

UPDATE: Added visualize_records to visualize multiple histories a29288c

This is useful for checking at what time it is valid between different histories such as associations. In the example below, it's easy to see that a department is valid in the employee's three histories.

irb(main):006:0> puts ActiveRecord::Bitemporal::Visualizer.visualize_records(employees)

                        | 2021-08-16 18:12:30.748
                        |         | 2021-11-01 16:08:28.857
                        |         | | 2021-11-04 10:26:47.052
                        |         | |       | 2022-01-04 16:05:27.859
                        |         | |       |                   | 9999-12-31 09:00:00.000
2021-08-16 18:40:32.432 |---------------------------------------|
                        |                                       |
2021-11-01 17:17:18.658 |---------|-----------------------------|
                        |         |                             |
2021-11-04 11:23:40.352 |         |-|---------------------------|
                        |         | |                           |
2022-01-04 16:42:01.277 |         | |-------|-------------------|
                        |         | |       |                   |
                        |         | |       |                   |
                        |         | |       |                   |
9999-12-31 09:00:00.000 |---------|-|-------|-------------------|

irb(main):007:0> puts ActiveRecord::Bitemporal::Visualizer.visualize_records(employees, [department])

                        | 2021-08-16 18:12:30.748
                        |         | 2021-11-01 16:08:28.857
                        |         | | 2021-11-04 10:26:47.052
                        |         | |       | 2022-01-04 16:05:27.859
                        |         | |       |                   | 9999-12-31 09:00:00.000
2021-08-16 18:40:32.432 |---------------------------------------|
                        |                                       |
2021-11-01 17:17:18.658 |---------|-----------------------------|
                        |         |                             |
2021-11-01 17:17:21.558 |         |-----------------------------|
                        |         |*****************************|
2021-11-04 11:23:40.352 |         |*****************************|
                        |         |*****************************|
2021-11-04 11:23:43.763 |         |-----------------------------|
                        |         | |                           |
2022-01-04 16:42:01.277 |         | |-------|-------------------|
                        |         | |       |                   |
9999-12-31 09:00:00.000 |---------|-|-------|-------------------|

osyo-manga · 2022-07-12T06:32:11Z

lib/activerecord-bitemporal/visualizer.rb

+          body.print("#{record.transaction_from.strftime('%F %T.%3N')} ", line: line)
+          if width > 0
+            if height > 0
+              body.print('|' + '-' * width + '|', line: line, column: column)


[nits] Changing '|' to '+' will make it a little easier to see.
https://github.com/kufu/activerecord-bitemporal/pull/94/files#diff-69d7b91cda642b52f976ad3f279e8d9aeb47b167a299f8426a6fc5d535c3aecfR88

before

| 2019-01-10 00:00:00.000 | | 2019-01-15 00:00:00.000 | | | 2019-01-20 00:00:00.000 | | | | 9999-12-31 00:00:00.000 2019-01-10 00:00:00.000 |---------------------------------------| | | 2019-01-15 00:00:00.000 |---------|-----------------------------| | | | | | | 2019-01-20 00:00:00.000 | |---------|-------------------| | | |*******************| | | |*******************| | | |*******************| | | |*******************| 9999-12-31 00:00:00.000 |---------|---------|-------------------|

after

| 2019-01-10 00:00:00.000 | | 2019-01-15 00:00:00.000 | | | 2019-01-20 00:00:00.000 | | | | 9999-12-31 00:00:00.000 2019-01-10 00:00:00.000 +---------------------------------------+ | | 2019-01-15 00:00:00.000 +---------+-----------------------------+ | | | | | | 2019-01-20 00:00:00.000 | +---------+-------------------+ | | |*******************| | | |*******************| | | |*******************| | | |*******************| 9999-12-31 00:00:00.000 +---------+---------+-------------------+

Fixed 9bafa9e

osyo-manga

Great! Thanks!!

sugamasao

great works for visualization 🎉

LGTM 👍

auto-assign bot requested review from osyo-manga and sugamasao June 23, 2022 10:44

osyo-manga reviewed Jun 23, 2022

View reviewed changes

wata727 force-pushed the bitemporal_visualizer branch 3 times, most recently from d3e64d8 to bc947fa Compare June 27, 2022 02:10

osyo-manga reviewed Jul 12, 2022

View reviewed changes

osyo-manga approved these changes Jul 12, 2022

View reviewed changes

sugamasao approved these changes Jul 13, 2022

View reviewed changes

Add bitemporal data structure visualizer

d3d538d

wata727 force-pushed the bitemporal_visualizer branch from 9bafa9e to d3d538d Compare July 13, 2022 02:59

wata727 merged commit 00774c9 into kufu:master Jul 13, 2022

wata727 deleted the bitemporal_visualizer branch July 13, 2022 03:02

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add bitemporal data structure visualizer #94

Add bitemporal data structure visualizer #94

wata727 commented Jun 23, 2022 •

edited

osyo-manga Jun 23, 2022

osyo-manga Jun 23, 2022

wata727 Jun 24, 2022 •

edited

osyo-manga Jun 24, 2022

osyo-manga commented Jul 4, 2022

wata727 commented Jul 4, 2022 •

edited

osyo-manga commented Jul 4, 2022

wata727 commented Jul 4, 2022

wata727 commented Jul 6, 2022

osyo-manga Jul 12, 2022

wata727 Jul 12, 2022

osyo-manga left a comment

sugamasao left a comment

	histories = record.class.ignore_valid_datetime.within_deleted.bitemporal_for(record)
	histories = record.class.ignore_bitemporal_datetime.bitemporal_for(record)

		valid_times = (histories.map(&:valid_from) + histories.map(&:valid_to)).sort.uniq
		transaction_times = (histories.map(&:transaction_from) + histories.map(&:transaction_to)).sort.uniq

Add bitemporal data structure visualizer #94

Add bitemporal data structure visualizer #94

Conversation

wata727 commented Jun 23, 2022 • edited

osyo-manga Jun 23, 2022

Choose a reason for hiding this comment

osyo-manga Jun 23, 2022

Choose a reason for hiding this comment

wata727 Jun 24, 2022 • edited

Choose a reason for hiding this comment

osyo-manga Jun 24, 2022

Choose a reason for hiding this comment

osyo-manga commented Jul 4, 2022

wata727 commented Jul 4, 2022 • edited

osyo-manga commented Jul 4, 2022

wata727 commented Jul 4, 2022

wata727 commented Jul 6, 2022

osyo-manga Jul 12, 2022

Choose a reason for hiding this comment

before

after

wata727 Jul 12, 2022

Choose a reason for hiding this comment

osyo-manga left a comment

Choose a reason for hiding this comment

sugamasao left a comment

Choose a reason for hiding this comment

wata727 commented Jun 23, 2022 •

edited

wata727 Jun 24, 2022 •

edited

wata727 commented Jul 4, 2022 •

edited