Preview #14

lokal-profil · 2017-06-08T11:52:52Z

Adds the -preview_file argument which, if provided, outputs a
preview of the changes to that file rather than making the actual
changes. Can be run together with the -simulate argument for
paranoid users.

Also:

Fix broken import in RBD
Fix novalue/somevalue handling in impact type
Fix single value in impact type
Implement set_common_values fully for RBD
Merge process_all_rbd and process_country_rbd in RBD
Fix minor bugs/typos

Task: T166673

@todo

Adds the `-preview_file` argument which, if provided, outputs a preview of the changes to that file rather than making the actual changes. Can be run together with the `-simulate` argument for paranoid users. Also: * Fix broken import in RBD * Fix novalue/somevalue handling in impact type * Fix single value in impact type * Fix minor bugs/typos @todo: * Add tests for PreviewItem * Handle/tasks for any @todos Task: [T166673](https://phabricator.wikimedia.org/T166673)

This allows us to drop the country parameter which was being passed around and to merge process_all and process_country

Per https://phabricator.wikimedia.org/T167388#3331814

Vesihiisi · 2017-06-13T06:20:06Z

WFD/PreviewItem.py

+
+
+class PreviewItem(object):
+    """Base bot to enrich Wikidata with info from WFD."""


Update this.

Vesihiisi · 2017-06-13T06:25:08Z

WFD/RBD.py

+    # @todo: T167662
+    def process_single_rbd(self, data, item):
+        """
+        Process a rbd (whether item exists or not).


an rbd; capitalize consistently as RBD since that's what's used in other places?

standardised RBD, SWB and WFD throughout.

Vesihiisi · 2017-06-13T06:26:58Z

WFD/swb_import.py

@@ -76,11 +80,11 @@ def set_common_values(self, data):
        """
        Set and validate values shared by every SWB in the dataset.

-        :param data: dict of all the SWB:s in the RBD
+        :param data: SWB component of thexml data


Vesihiisi · 2017-06-13T06:28:59Z

WFD/swb_import.py

@@ -95,8 +99,9 @@ def process_all_swb(self, data):

        Only increments counter when an swb is updated.

-        :param data: dict of all the swb:s in the RBD
+        :param data: list of all the swb:s in the RBD


But since you then run helpers.listify on it, it means that something else than a list can be passed here?

This is due to the source data changing one item lists to just the item. Clarifying.

Vesihiisi · 2017-06-13T06:32:13Z

WFD/PreviewItem.py

+            else:
+                raise ValueError(
+                    'Sorry but "{}" is not a recognized special '
+                    'value/snacktype.'.format(wd_id))


Vesihiisi · 2017-06-13T06:48:20Z

WFD/RBD.py

@@ -43,9 +47,11 @@
 class RbdBot(WfdBot):
    """Bot to enrich/create info on Wikidata for RBD objects."""

-    def __init__(self, mappings, year, new=False, cutoff=None):
+    def __init__(self, mappings, year, new=False, cutoff=None,
+                 preview_file=None):


Maybe use some default path if none given? At least for me it was the most comfortable option, and could be used after moving this to Wikidatastuff, e.g. using the format name of dataset + timestamp.

The issue here is that I use -preview_file to control if a preview is generated rather than an upload. If I set a default then I would need to separate out the trigger mechanism into one more commandline argument.

Vesihiisi · 2017-06-13T06:50:49Z

WFD/swb_import.py

@@ -311,6 +335,8 @@ def validate_indata(data, mappings):
    """
    # validate mapping of each <surfaceWaterBodyCategory> and
    # <swSignificantImpactType>
+    pywikibot.output('Target file containes {} entries, validating...'.format(


Vesihiisi · 2017-06-13T07:55:59Z

WFD/PreviewItem.py

+        """
+        Create a label(s) preview.
+
+        The first label is italicised (to indicate it is the preferred label


As for the presentation of labels / aliases -- the current format works, but I think putting every label / alias on a separate line would make it even more visually clear. I'm saying this with the future generalization in Wikidatastuff in mind (multiple or long labels), but even here when the labels are long it can get hard to read:

en: 2. Bothnian Sea (International drainage basin Trondelagsfylkene - Sweden) | SE1102

If you happen to have an alias that's equally long, or two aliases, they might get hard to parse when they're all on one line.

I broke this out as T167791. Separating per line is probably the best solution but I want to ensure it works (without swamping the preview page) even when there are multiple languages.

Vesihiisi · 2017-06-13T08:00:53Z

WFD/PreviewItem.py

+        if self.item:
+            return PreviewItem.make_wikidata_template(self.item)
+        else:
+            return 'New item created'


Not sure if this isn't visual noise. Leaving this blank works as well, and is easier to parse by the reader.

You might be right but I'm loath to leave it completely blank.Changed it to –. Would that work?

Vesihiisi · 2017-06-13T08:02:06Z

WFD/PreviewItem.py

+        """
+        Make a wikidata template for items and properties.
+
+        :param item: a Q/P prefixed item/property id or a ItemPage/PropertyPage


The parameter is called wd_entry here.

This ensures e.g. P17 comes before P137.

lokal-profil · 2017-06-13T13:20:51Z

Also updated https://www.wikidata.org/wiki/User:AndreCostaWMSE-bot/WFD/preview#RBD

lokal-profil added 8 commits June 8, 2017 12:28

Minor CI happiness

4a42754

clarify contents

b3939b3

Implement set_common_values fully for RBD

5a444ea

This allows us to drop the country parameter which was being passed around and to merge process_all and process_country

More whitespace

24bf358

Per https://phabricator.wikimedia.org/T167388#3331814

Fixing/linking todos and making ref optional

e2302e9

Tests and related fixes

8410a41

Make CI happy

adec887

lokal-profil requested a review from Vesihiisi June 12, 2017 18:40

Vesihiisi reviewed Jun 13, 2017

View reviewed changes

lokal-profil added 5 commits June 13, 2017 14:28

First batch of review comments

d6488bc

Second half of review implementation

3e8750b

That was not py3

85f10ff

Ensure order of labels/descriptions/protoclaims are sorted

36dc64c

Sort protoclaims by the numeric part of the Pid

1d147fa

This ensures e.g. P17 comes before P137.

Vesihiisi approved these changes Jun 13, 2017

View reviewed changes

lokal-profil merged commit 002c103 into master Jun 13, 2017

lokal-profil deleted the preview branch June 13, 2017 19:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preview #14

Preview #14

lokal-profil commented Jun 8, 2017 •

edited

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017 •

edited

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

Vesihiisi Jun 13, 2017

lokal-profil Jun 13, 2017

lokal-profil commented Jun 13, 2017



		class PreviewItem(object):
		"""Base bot to enrich Wikidata with info from WFD."""

Preview #14

Preview #14

Conversation

lokal-profil commented Jun 8, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lokal-profil Jun 13, 2017 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lokal-profil commented Jun 13, 2017

lokal-profil commented Jun 8, 2017 •

edited

lokal-profil Jun 13, 2017 •

edited