`Graph.subjects` and `Graph.predicates` should return only unique values

[mertzjames]

When calling the Graph method subjects or predicates, the generator should only return unique values. For example:

from rdflib import Graph
graph = Graph()
graph.parse("https://www.w3.org/People/Berners-Lee/card")
list(graph.subjects())

returns the following:

[rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.BNode('N170b6415b06f45c98cb7a09398bbe818'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('http://www.w3.org/data#W3C'),
 rdflib.term.URIRef('http://www.ecs.soton.ac.uk/~dt2/dlstuff/www2006_data#panel-panelk01'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('http://wiki.ontoworld.org/index.php/_IRW2006'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('http://www.w3.org/2011/Talks/0331-hyderabad-tbl/data#talk'),
 rdflib.term.BNode('Nb0ee70dcdcc042609d1dde9df6b9efd5'),
 rdflib.term.BNode('Nebae90d841e441adbf72a3836347c354'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.URIRef('http://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/breadcrumbs/blog/4'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/data#DIG'),
 rdflib.term.URIRef('https://www.w3.org/DesignIssues/Overview.html'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/2007/01/camp/data#course'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.BNode('Nebae90d841e441adbf72a3836347c354'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.BNode('Nce03d86469d64deb945313ef31a8e661'),
 rdflib.term.BNode('Nce03d86469d64deb945313ef31a8e661'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/DesignIssues/Overview.html'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('http://www.w3.org/2011/Talks/0331-hyderabad-tbl/data#talk'),
 rdflib.term.URIRef('http://www.w3.org/2000/10/swap/data#Cwm'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/breadcrumbs/blog/4'),
 rdflib.term.URIRef('http://wiki.ontoworld.org/index.php/_IRW2006'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.URIRef('http://www.ecs.soton.ac.uk/~dt2/dlstuff/www2006_data#panel-panelk01'),
 rdflib.term.BNode('N170b6415b06f45c98cb7a09398bbe818'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('Nce03d86469d64deb945313ef31a8e661'),
 rdflib.term.BNode('Nb0ee70dcdcc042609d1dde9df6b9efd5'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('Nebae90d841e441adbf72a3836347c354'),
 rdflib.term.BNode('N713bca5215e14aa4ae1a3dfca525ce37'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N713bca5215e14aa4ae1a3dfca525ce37'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/breadcrumbs/blog/4'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/2005/ajar/ajaw/data#Tabulator'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0')]

I would expect it to return something more along the lines of:

[rdflib.term.URIRef('http://www.ecs.soton.ac.uk/~dt2/dlstuff/www2006_data#panel-panelk01'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/data#DIG'),
 rdflib.term.URIRef('http://www.w3.org/2000/10/swap/data#Cwm'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/2005/ajar/ajaw/data#Tabulator'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card'),
 rdflib.term.URIRef('http://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.BNode('N170b6415b06f45c98cb7a09398bbe818'),
 rdflib.term.BNode('N1d372c7ed22e4de5a167177739cc29a0'),
 rdflib.term.URIRef('http://wiki.ontoworld.org/index.php/_IRW2006'),
 rdflib.term.BNode('N112689303c124871942e42a7ded52e64'),
 rdflib.term.URIRef('https://www.w3.org/DesignIssues/Overview.html'),
 rdflib.term.BNode('N713bca5215e14aa4ae1a3dfca525ce37'),
 rdflib.term.URIRef('http://www.w3.org/data#W3C'),
 rdflib.term.BNode('Nce03d86469d64deb945313ef31a8e661'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/2007/01/camp/data#course'),
 rdflib.term.BNode('Nebae90d841e441adbf72a3836347c354'),
 rdflib.term.URIRef('http://www.w3.org/2011/Talks/0331-hyderabad-tbl/data#talk'),
 rdflib.term.BNode('Nb0ee70dcdcc042609d1dde9df6b9efd5'),
 rdflib.term.URIRef('https://www.w3.org/People/Berners-Lee/card#i'),
 rdflib.term.URIRef('https://timbl.com/timbl/Public/friends.ttl'),
 rdflib.term.URIRef('http://dig.csail.mit.edu/breadcrumbs/blog/4')]

While you could technically get the list of unique subjects using list(set(graph.subjects())) it would be nice if subjects did this automatically.

[mertzjames]

One easy way to solve this would be to do the following:

    def subjects(self, predicate=None, object=None):
        """A generator of subjects with the given predicate and object"""
        subjects = []
        for s, p, o in self.triples((None, predicate, object)):
            if s not in subjects:
                subjects.append(s)
                yield s

[nicholascar]

Still an issue, flagging for 6.0.0

[white-gecko]

I think adding all subjects to a list while iterating will wast a lot of memory which can become problematic for big graphs.

On the other hand it would also be cool to have a method nodes that would return a distinct list of all subjects and objects, or a distinct list of all subjects, predicates, and objects.

[tgbugs]

I think that any implementation for this will be O(n) in memory usage because it will have to maintain a 'seen' list (or equivalent). Thus, if it is implemented it shouldn't be the default behavior because it will break all cases where the backend store won't fit in memory. It might be possible for stores to indicate whether they support returning unique subjects, predicates, objects, etc. that might enable this behavior without introducing the memory issues that the naive implementation would likely impose.

[white-gecko]

So, do we just want to say, invalid for this issue?

[mertzjames]

How have you currently implemented the generator? Within there, you can add the logic to basically skip already seen fields. I think this would actually save time for the user if I'm not mistaken.

[LvanWissen]

What if the SPARQLStore has a query limit (e.g. 10000)? Transforming the results from the generator into a set could theoretically mean that results are left out, because they have not been returned from the endpoint. Implementing a DISTINCT in the select query prevents this.

[nilay77]

I have made a PR #1068 to solve this issue. Please have a look.

[metazool]

set([str(url) for url in graph.subjects()]) is fine for everyday use on non-enormous graphs, the issue with the associated closed PR with the explicit unique parameter is that you are losing the advantage of the generator? It looked promising all the same