Add a sanity function for extracting multiple capturing groups at a time

[vkarak]

This is something that has been requested to me some times, so I guess it would make sense to consider it. The question is how the returned value is meant to be treated. A straightforward return value would be to return a list of tuples of the extracted values for the different capturing groups or would it make sense to return a list-like data structure that would allow, for example, to extract all the elements of a single capturing group at once? @jgphpc How you would use such a function?

[jgphpc]

My usage would be as follows. Typical stdout looks like this:

Table 1:  Load Balance with MPI Message Stats

Level,Samp%,Samp,Group/PE=[mmm]
0,100.0%,1247.4,Total
1,76.2%,950.2,USER
2,81.4%,1015.0,USER/pe.56
2,79.0%,985.0,USER/pe.69
2,1.6%,20.0,USER/pe.94
1,18.2%,226.8,MPI
2,94.4%,1178.0,MPI/pe.95
2,15.3%,191.0,MPI/pe.38
2,12.0%,150.0,MPI/pe.20
1,5.4%,67.3,ETC
2,6.7%,83.0,ETC/pe.17
2,5.4%,67.0,ETC/pe.29
2,3.3%,41.0,ETC/pe.95

To extract % and pe, i would get 2 lists like this:

        regex = r'^\d+,\S+,(?P<samples>\S+),ETC/pe.(?P<pe>\d+)$'
        res_etc_samples = sn.extractall(regex, stdout, 'samples', float)
        res_etc_pe = sn.extractall(regex, stdout, 'pe')

It would be kind of nice to be able to:

regex = r'^(?P<key>\d+)  (?P<value>\d+)$'
result_d = sn.extractall(regex, self.stdout, 'key', 'value', float, float)

then use it as:

max(result_d.items(), key=operator.itemgetter(1))[0]

For instance:

>>> res = {}
>>> res['pe.17'] = 6.7
>>> res['pe.29'] = 5.4
>>> res['pe.95'] = 3.3
>>> import operator
>>> max(res.items(), key=operator.itemgetter(1))[0]
'pe.17'

Another way is to do this: create a dict from the 2 lists dict(zip(L1,L2))

[teojgo]

@vkarak I am thinking that we could have an extractall_multiple kind of function that could get a list of tuples (group, func) which would extract the group and apply the conversion function to it. It would return either a list or a dict with keys the capturing groups.

[vkarak]

@teojgo Can you provide some usage examples, so that I can visualize the interface you propose. It's not only what to be returned, but also what to be passed to the function.

[teojgo]

For example to extract what @jgphpc wants from here:

Table 1:  Load Balance with MPI Message Stats

Level,Samp%,Samp,Group/PE=[mmm]
0,100.0%,1247.4,Total
1,76.2%,950.2,USER
2,81.4%,1015.0,USER/pe.56
2,79.0%,985.0,USER/pe.69
2,1.6%,20.0,USER/pe.94
1,18.2%,226.8,MPI
2,94.4%,1178.0,MPI/pe.95
2,15.3%,191.0,MPI/pe.38
2,12.0%,150.0,MPI/pe.20
1,5.4%,67.3,ETC
2,6.7%,83.0,ETC/pe.17
2,5.4%,67.0,ETC/pe.29
2,3.3%,41.0,ETC/pe.95

He would use:

regex = r'^(?P<key>\d+)  (?P<value>\d+)$'
result_d = sn.extractall_multiple(regex, self.stdout, [('key', float), ('value', float)])

And would get either a dict with two keys key and value where each list would contain the corresponding float values, or you would get a list of tuples where each tuple will contain the key, value matches converted to float.

[vkarak]

This is the current API:

def extractall(patt, filename, tag=0, conv=None, encoding='utf-8'):

The natural extension would be to allow tag and/or conv to be iterables, in which case we return a list of tuples such as this one [(val0_of_tag0, val0_of_tag1, ...), (val1_of_tag0, val1_of_tag1, ...)]. This change has to be applied to the whole family of extract* functions. I'm not so much in favor of providing another function, because that would imply that we should provide additional functions for every extract* variant. In any case, we need to properly check the arguments.

[teojgo]

In this case, we would have to support the following three cases:

• Single tag, single conv -> As it is now
• Iterable tag, single conv -> The single conv would be applied for all the tags
• Iterable tag, iterable conv -> They should have the same length where every tag will have a matching conversion function

Is this ok?

[vkarak]

Why not don't we internally convert tag and conv to lists and and always extend conv to match the size of tag? Also, depending on what we got (lists or simple values), we adapt the return value as well.