Join-by-nearest can mismatch field names when run in a script/plugin.

[tetrakai1]

What is the bug or the crash?

See the MWE below.

When the order of FIELDS_TO_COPY is different from the order in the input layer, Field1_2 now has value D (rather than C) in the output layer.

This will not be seen when run through the config dialog because the correct order is enforced, however it can be a problem as part of a script/plugin when used on new input data. Eg, QuickOSM will return fields in different order depending on the exact query.

These appear to be the responsible lines:

• Field names/indices added in same order as found in the fieldsToCopy parameter. Eg, index 2 can be before index 1.

for ( const QString &field : fieldsToCopy )
{
  const int index = input2->fields().lookupField( field );
  if ( index >= 0 )
  {
    fields2Indices << index;
    outFields2.append( input2->fields().at( index ) );
  }
}

• Attributes (field values) added sequentially. Ie, in same order as found in the input layer.

QgsAttributes attributes;
const int attributeCount = f.attributeCount();
for ( int j = 0; j < attributeCount; ++j )
{
  if ( ! fields2Indices.contains( j ) )
    continue;
  attributes << f.attribute( j );
}
input2AttributeCache.insert( f.id(), attributes );

https://github.com/qgis/QGIS/blob/master/src/analysis/processing/qgsalgorithmjoinbynearest.cpp

Steps to reproduce the issue

MWE:

import pandas as pd

# Create a point layer containing a single point
def new_layer(name, attr1, attr2, X, Y):
    layer    = QgsVectorLayer("Point", name, "memory")
    provider = layer.dataProvider()
    provider.addAttributes([QgsField("Field1", QVariant.String), 
                            QgsField("Field2", QVariant.String)])

    newfeat = QgsFeature(layer.fields())
    newfeat.setAttributes([attr1, attr2])

    newfeat.setGeometry(QgsGeometry.fromPoint(QgsPoint(X, Y)))

    provider.addFeature(newfeat)
    layer.updateFields()

    return layer

# Create two layers
layer1 = new_layer("layer1", "A", "B", 0, 0)
layer2 = new_layer("layer2", "C", "D", 1, 1)

# Run join-by-nearest algorithm
fields_to_copy = [['Field1','Field2'], ['Field2','Field1']]
for fields in fields_to_copy:
    alg_params = {'DISCARD_NONMATCHING' : False, 
                  'FIELDS_TO_COPY'      : fields, 
                  'INPUT'               : layer1, 
                  'INPUT_2'             : layer2, 
                  'MAX_DISTANCE'        : None, 
                  'NEIGHBORS'           : 1, 
                  'OUTPUT'              : 'TEMPORARY_OUTPUT', 
                  'PREFIX'              : '' }
    
    joined_layer = processing.run('native:joinbynearest', alg_params)['OUTPUT']
    
    res = {"name" : joined_layer.fields().names(),
           "value" : next(joined_layer.getFeatures())}

    print()
    print({"Input Field Order": fields})
    print(pd.DataFrame(res))

Results:

{'Input Field Order': ['Field1', 'Field2']}
        name     value
0     Field1         A
1     Field2         B
2   Field1_2         C
3   Field2_2         D
4          n         1
5   distance  1.414214
6  feature_x       0.0
7  feature_y       0.0
8  nearest_x       1.0
9  nearest_y       1.0

{'Input Field Order': ['Field2', 'Field1']}
        name     value
0     Field1         A
1     Field2         B
2   Field2_2         C
3   Field1_2         D
4          n         1
5   distance  1.414214
6  feature_x       0.0
7  feature_y       0.0
8  nearest_x       1.0
9  nearest_y       1.0

Versions

QGIS version	3.40.3-Bratislava
QGIS code revision	2a274ab7754
Libraries
Qt version	5.15.3
Python version	3.10.12
GDAL/OGR version	3.4.1
PROJ version	8.2.1
EPSG Registry database version	v10.041 (2021-12-03)
GEOS version	3.10.2-CAPI-1.16.0
SQLite version	3.37.2
PDAL version	2.3.0
PostgreSQL client version	14.15 (Ubuntu 14.15-0ubuntu0.22.04.1)
SpatiaLite version	5.0.1
QWT version	6.1.4
QScintilla2 version	2.11.6
OS version	Linux Mint 21.3
Active Python plugins
QuickOSM	2.3.2
plugin_reloader	0.17
GetNameCandidates	0.1
processing	2.12.99
MetaSearch	0.3.6
grassprovider	2.12.99
db_manager	0.1.20

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

No response

[alexbruy]

Probably related ticket #48451.

[tetrakai1]

Probably related ticket #48451.

Yes, that looks like the exact same thing. My workaround was to sort the FIELDS_TO_COPY parameter before running it. Eg, if you just need name and ref fields:

fieldnames = input2_layer.fields().names()
if fieldnames.index('ref') > fieldnames.index('name'):
   fields_to_copy = ['name', 'ref']
else:
   fields_to_copy = ['ref', 'name']

alg_params = {
   'DISCARD_NONMATCHING': True,
   'FIELDS_TO_COPY'     : fields_to_copy,
   'INPUT'              : input1_layer,
   'INPUT_2'            : input2_layer,
   'MAX_DISTANCE'       : 100,
   'NEIGHBORS'          : 50,
   'PREFIX'             : '',
   'OUTPUT'             : QgsProcessing.TEMPORARY_OUTPUT
}
res = processing.run('native:joinbynearest', alg_params)

A more generic version of that may be the easiest fix.

[roya0045]

Probably related ticket #48451.

Different alg, but you are right, the same logic issue seems to be present in both. My main guess are those line pausing problem as the loop check in order of appearance in the joined feature and join them if they are contained, ignoring the listed order of the field provided in both.

QGIS/src/analysis/processing/qgsalgorithmjoinbynearest.cpp

Lines 206 to 211 in c6edab4

	for ( int j = 0; j < attributeCount; ++j )
	{
	if ( ! fields2Indices.contains( j ) )
	continue;
	attributes << f.attribute( j );
	}

QGIS/src/analysis/processing/qgsalgorithmjoinbyattribute.cpp

Lines 195 to 199 in c6edab4

	for ( int j = 0; j < attributeCount; ++j )
	{
	if ( !fields2Indices.contains( j ) )
	continue;
	attributes << feat.attribute( j );

The fix should be similar.

I could look into making a PR if you're not already on it.