Extend Wikidata extension to support arbitrary Wikibase instances #1640

susannaanas · 2018-06-06T11:11:40Z

Roadmap for this feature (copied from @wetneb's Aug 13, 2018 comment #1640 (comment)):

Work with the MediaWiki developers to have the MediaWiki API endpoint expose configuration of extensions (such as Wikibase, Quality Constraints). This is a long term goal, I expect it will take ages so it should not be blocking. Task: https://phabricator.wikimedia.org/T155155
Draft and document an initial version of the manifest format. The format should include versioning so that we can switch to something lighter once MediaWiki exposes more info. Task: https://phabricator.wikimedia.org/T197588
Write Java classes to represent these manifests (with versioned implementation), with serialization and storage in OR's preferences. Each Wikibase instance should be linked to a default reconciliation service, used in the schema UI. The registry key should be something stable, such as the MediaWiki API endpoint or the manifest URI.
Create UI to list known instances, add a new one by providing the URL of a manifest.
Change schema serialization to include a mention of the Wikibase instance (not the full manifest). When not provided, fallback to Wikidata.
Percolate the instance through schema evaluation, quality assurance and editing.
Change the UI of the schema editor to let the user choose which instance to use.
Cleanup mentions to Wikidata -> Wikibase in the source, UI and translations.

Questions to be resolved:

which key to use for the instances? Because the workflows must be reproducible, the manifest URL must appear in the operations, so using only the MediaWiki API URL does not seem to be enough. But using the manifest URL means pinning down the manifest version (including constraint configuration, and other things that might change fairly regularly).
do we store only one overlay model or parallel schema for different instances? leaning to the former for simplicity of the UI.

wetneb · 2018-06-07T11:39:24Z

Right now it does not, but yes it should!

thadguidry · 2018-06-07T12:07:23Z

@susannaanas Is it a super urgent need ? Curious, Do you know of how many other folks that would benefit from this enhancement other than you ?

susannaanas · 2018-06-07T12:12:01Z

I would find use for it immediately, but I would not describe it as super urgent. There is a growing number of Wikibase instances, listed in this Wikibase https://wikibase-registry.wmflabs.org/. I could not do it myself, but I wonder if the component could be configured by a skilled developer to point to another Wikibase?

wetneb · 2018-06-08T08:50:31Z

@susannaanas it's a significant amount of work to do it properly. The first step would be to make the reconciliation interface run on any Wikibase, then create a notion of "profile of Wikibase instance" which contains all the information necessary to run the extension (MediaWiki API, reconciliation API, settings of WikibaseQualityConstraints, EditGroups…) and then forward that everywhere where this is needed.

susannaanas · 2018-06-08T13:48:53Z

Sounds much more complex than I imagined!

BTW, I think also Structured Data for Commons qualifies as one such Wikibase.

wetneb · 2018-06-09T09:03:01Z

sure, and that will need even more adaptations because the further tweaks in the data model (ids of the pages to edit will have a new format, but properties ids and items will be borrowed from Wikidata…)

wetneb · 2018-06-10T08:32:26Z

for reference, this is a feature request I have heard quite often (especially at the Wikimedia Hackathon) so it would be good to keep this in mind as a mid-term goal.

wetneb · 2018-08-07T09:55:31Z

@susannaanas it should now be easier to run the reconciliation interface for other Wikibase instances. I have added some docs and a Dockerfile at https://github.com/wetneb/openrefine-wikibase

susannaanas · 2018-08-07T10:00:14Z

Wow, we'll start testing it (soonish)!

Thanks!

wetneb · 2018-08-13T10:59:28Z

Roadmap for this feature:

[copied to main issue body above]

Questions to be resolved:

which key to use for the instances? Because the workflows must be reproducible, the manifest URL must appear in the operations, so using only the MediaWiki API URL does not seem to be enough. But using the manifest URL means pinning down the manifest version (including constraint configuration, and other things that might change fairly regularly).
do we store only one overlay model or parallel schema for different instances? leaning to the former for simplicity of the UI.

ettorerizza · 2019-07-25T09:57:38Z

Interesting diagram to better understand the current situation: https://github.com/HeardLibrary/linked-data/blob/master/wikibase/interaction-diagram.png

msaby · 2019-08-06T09:50:50Z

By the way : French national agencies will use wikidata for their prototype of national authorities file. So it could be useful for us

wetneb · 2019-08-06T10:08:32Z

Yes, and OCLC is experimenting with WIkibase too: https://www.oclc.org/research/publications/2019/oclcresearch-creating-library-linked-data-with-wikibase-project-passage.html

belett · 2020-01-17T15:51:22Z

Hi any update on this?

wetneb · 2020-01-21T21:13:44Z

No, this is still in the same state.

As much as I would like to solve this issue (and many others related to the Wikidata extension), I cannot spend all my time on Wikidata integration in OpenRefine - many other aspects of OpenRefine need my attention too! My hope was that the Wikibase community would get involved in this, but so far this has not really happened yet.

If anyone wants to work on this, I am available to help them.

) Closes #1640. All Wikibase-dependent parameters, which were previously hard-coded for Wikidata, are now described in a JSON manifest. The manifest is currently constructed by hand, but, in the future, will hopefully be published by each Wikibase instance at a standard location. * setup the manifest framework * add dependency mechanism to scrutinizers & update tests * add json creators to constraint entities * adapt the backend (units tests are to be updated) * remove the call to prepareDependencies() in the constructor * update code according to review feedback * update scrutinizers tests * fix typo & update ConstraintsV1 * log if a scrutinizer is skipped * update versioning handling in the backend * correct the order of "actual" and "expected" for assertEquals method * use regex to check manifest versions * 1. add wikibase-manager.js, wikibase-dialog.js, etc. 2. move dialog/schema-alignment-dialog.js -> schema-alignment.js 3. remove unused schema-alignment-dialog.html 4. change most mentions of "Wikidata" to "Wikibase" * support saving cookies for different Wikibases & fix LoginCommandTest * fix schema related tests * removed unused WikibaseCredentials * include MediaWiki API endpoint in the schema * fetch language codes for different Wikibases * fix lgtm-bot alerts * keep a connection map (MediaWiki API endpoint => Connection) in ConnectionManager * simplify the constraint configurations of the manifest and remove lots of unnecessary code. * add slash to the end of mediawiki.root * add manifest schema and use ajv to validate the manifest * remove JSONP support (Wikibase manifest host should support CORS) * save manifests on manifest update * add unit tests for Manifest * include the exception in logger.error() method to make it easier to debug * include the message of ManifestException when calling respondError * test multiple connections * test no manifest & test invalid manifest * adapt manage-account-dialog.js to support multiple Wikibase connections * update instance/subclass of related translations * beautify import-schema-dialog.html * use "${lang}" variable in the reconciliation service endpoint of the manifest * adapt schema-alignment.js after introducing "${lang}" variable in the reconciliation service endpoint * use WikibaseManager.getSelectedWikibaseApi() in SchemaAlignment._getPropertyType * replace more mentions of "Wikidata" to "Wikibase" * use WikibaseManager.getSelectedWikibaseApi() in previewrenderer.js * support fetching language codes of different Wikibases in the frontend * skip EditInspector if missing 'property_constraint_pid' in the manifest * improve unit tests for fetching lang codes * skip scrutinizers depending on fetcher if 'property_constraint_pid' is missing in the manifest * make sure the schema alignment panel is set up before rendering * fix preview bug * add getters of "instance of" and "subclass of" to the Manifest interface and use them in NewItemScrutinizer * fix hardcode for Wikidata in WbItemVariable * rename 'entity_prefix' to 'site_iri' and move it from 'manifest.wikibase.properties' to 'manifest.wikibase' * include oauth configurations in the manifest & support logging in with owner-only consumer for Wikibases with the OAuth extension * correct schema fallback logic * select default wikibase according to the saved schema * include maxlag in the manifest * [backend] move maxlag setting from preferences to request parameter * support setting maxlag when uploading edits * rename "Manage Wikibase" to "Select Wikibase instance" and localize it * fix manifest updating bug * include EditGroups in the manifest * add the reconciliation service from the manifest to standard services if it's not present yet when adding a new manifest * update according to review feedback 1. use inherited color variable 2. rename 'gridwroks' to 'openrefine' 3. remove unnecessary 'async: true' 4. add 'format: url' validation to urls to the schema * rename 'wikibasePrefix' to 'siteIri'

* Take snapshot of docs for version 3.4 * Versioning for docs of the cross function, for #2504 * Document 'Store archive file' option (#1963) * Remove unsupported preference from 3.4 docs (#2624) * Mention that forEach works on JSON objects (#3149) * Remove wholeText from 3.4 docs (#3180) * Document -H, /H CLI options (#3288) * Migrate Wikibase documentation from Wikidata (#1640) * Miscellanous, copy-editing

wetneb added the Type: Feature Request Identifies requests for new features or enhancements. These involve proposing new improvements. label Jun 7, 2018

wetneb added the wikibase Related to wikidata/wikibase integration label Dec 11, 2018

wetneb mentioned this issue Aug 30, 2019

Add support for editing structured data on Wikimedia Commons #2144

Closed

wetneb mentioned this issue Feb 3, 2020

Make maxlag configurable in Wikidata extension #2304

Closed

wetneb added the gsoc/outreachy Projects proposed for internships. Please hold back from these tasks if you are not elligible. label Mar 29, 2020

tfmorris assigned afkbrb May 4, 2020

tfmorris changed the title ~~Can OpenRefine Wikidata extension be made to work with other Wikibase instances?~~ Extend Wikidata extension to support arbitrary Wikibase instances May 4, 2020

afkbrb mentioned this issue Jun 25, 2020

Extend Wikidata extension to support arbitrary Wikibase instances #2810

Merged

thadguidry mentioned this issue Aug 12, 2020

EPIC - Allow opt-in to a central Reconciliation Service Registry #3071

Closed

6 tasks

tfmorris closed this as completed in #2810 Aug 22, 2020

thadguidry added this to the 3.5 milestone Aug 22, 2020

wetneb mentioned this issue Apr 24, 2021

Documentation versioning #3847

Merged

16 tasks

wetneb added a commit to wetneb/OpenRefine that referenced this issue Apr 25, 2021

Migrate Wikibase documentation from Wikidata (OpenRefine#1640)

3555f14

tfmorris mentioned this issue Jul 23, 2024

Add escape() function options to escape URL path and fragment #6715

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extend Wikidata extension to support arbitrary Wikibase instances #1640

Extend Wikidata extension to support arbitrary Wikibase instances #1640

susannaanas commented Jun 6, 2018 •

edited by afkbrb

Loading

wetneb commented Jun 7, 2018

thadguidry commented Jun 7, 2018

susannaanas commented Jun 7, 2018

wetneb commented Jun 8, 2018

susannaanas commented Jun 8, 2018

wetneb commented Jun 9, 2018

wetneb commented Jun 10, 2018

wetneb commented Aug 7, 2018

susannaanas commented Aug 7, 2018

wetneb commented Aug 13, 2018 •

edited by tfmorris

Loading

ettorerizza commented Jul 25, 2019

msaby commented Aug 6, 2019

wetneb commented Aug 6, 2019 •

edited

Loading

belett commented Jan 17, 2020

wetneb commented Jan 21, 2020

Extend Wikidata extension to support arbitrary Wikibase instances #1640

Extend Wikidata extension to support arbitrary Wikibase instances #1640

Comments

susannaanas commented Jun 6, 2018 • edited by afkbrb Loading

wetneb commented Jun 7, 2018

thadguidry commented Jun 7, 2018

susannaanas commented Jun 7, 2018

wetneb commented Jun 8, 2018

susannaanas commented Jun 8, 2018

wetneb commented Jun 9, 2018

wetneb commented Jun 10, 2018

wetneb commented Aug 7, 2018

susannaanas commented Aug 7, 2018

wetneb commented Aug 13, 2018 • edited by tfmorris Loading

ettorerizza commented Jul 25, 2019

msaby commented Aug 6, 2019

wetneb commented Aug 6, 2019 • edited Loading

belett commented Jan 17, 2020

wetneb commented Jan 21, 2020

susannaanas commented Jun 6, 2018 •

edited by afkbrb

Loading

wetneb commented Aug 13, 2018 •

edited by tfmorris

Loading

wetneb commented Aug 6, 2019 •

edited

Loading