Filter data by draft status

[movermeyer]

What are you trying to accomplish?

Fixes #73.

Throughout ruby-cldr, there were draft? calls in some places, but not others.

What approach did you choose and why?

I added a new --draft-status CLI option that allows users to specify the minimum draft status that they want all exported data to have.

Instead of each area of the codebase needing to know to check draft? all the time, all access to the data is done through a new DataFile class, which filters the data by draft status transparently. This ensures that we are not missing places where we need to be doing the filtering.

The default minimum draft status of contributed matches the status needed for inclusion into Unicode's ICU (and consequently what cldr-json exports).

What should reviewers focus on?

• DraftStatus is effectively an enum. Is there a better way to define these in pure Ruby?
• I use a new class variable to store the "global" minimum_draft_status. It was a lot nicer than passing a new minimum_draft_status parameter around everywhere. I feel mostly OK about it.

There is a similar issue with the alt attribute, but that will be handled as part of #125.

The impact of these changes

Users can now control what the minimum draft status to accept data from.

Less data exported as you increase your minimum draft status:

	v34 exported data size (bytes)	v37
Before this PR (main)	28903804 (27.5648 MiB)	30827560 (29.3995 MiB)
--draft-status=unconfirmed	29095943 (27.7481 MiB)	31018722 (29.5818 MiB)
--draft-status=provisional	28741819 (27.4103 MiB)	30671275 (29.2504 MiB)
--draft-status=contributed	28556418 (27.2335 MiB)	30539281 (29.1245 MiB)
--draft-status=approved	21065654 (20.0898 MiB)	22348295 (21.313 MiB)

(Computed using find . -type f -exec ls -l {} \; | awk '{sum += $5} END {print sum}', since MacOS' du command doesn't have a bytes flag)

Testing

thor cldr:download --version=37

Find an entry in the vendor directory that has draft=provisional, for example:

vendor/cldr/common/main/el.xml:

<currency type="ALK">
  <displayName count="one" draft="provisional">Παλαιό λεκ Αλβανίας</displayName>
  <displayName count="other" draft="provisional">Παλαιά λεκ Αλβανίας</displayName>
</currency>

Export with different minimum draft statuses and compare the results:

thor cldr:export --draft-status=provisional --target=provisional --components=Currencies --locales=el
thor cldr:export --draft-status=contributed --target=contributed --components=Currencies --locales=el
diff -r provisional contributed

And see that the diffs are the draft=provisional entries.

[movermeyer]

Most of this comment has been moved to DataFile#merge

The method itself has been rewritten to use reduce, which is cleaner IMO.

[trishrempel]

Looks good!

[trishrempel]

I'm wondering if this should set a default if not provided, since it will throw an exception if not set.

[movermeyer]

I think I prefer for it to explode, since it indicates that something is not working as expected. 🤷

[trishrempel]

You might want to not set the class variable explicitly by default, so you can see when an exception is thrown in tests when it's being unexpectedly used before it's been set.

You could set it explicitly for tests that expect it to already be set, or mock it and explicitly assert that it was used.

[movermeyer]

I think I want it to default for everything except for the tests that exercise the draft status filtering code. I've broken the tests in data_file_test.rb into two TestCases, and I override the defaulting in the one that is testing the draft status filtering. 👍