PGtokenizer is ignoring last token if last token is null or blank

[chaluvadi286]

Describe the issue
PGtokenizer is ignoring last token if last token is blank or NULL. For eg- (1,2EC1830300027,1,,) then it should return 5 tokens but it
is returning 4 tokens only. Sample code -
public static void main(String[] args) throws Exception {
String objectStr = "(1,2EC1830300027,1,,)";
PGtokenizer token = new PGtokenizer(PGtokenizer.removePara(objectStr), ',');
for (int i = 0; i < token.getSize(); i++) {
System.out.println(i + " = " + token.getToken(i) );
}
}

Output -
0 = 1
1 = 2EC1830300027
2 = 1
3 =

Driver Version?
postgresql-42.2.14.jar
Java Version?
1.8
OS Version?
Windows 10
PostgreSQL Version?
12
To Reproduce
Steps to reproduce the behaviour:
public static void main(String[] args) throws Exception {
String objectStr = "(1,2EC1830300027,1,,)";
PGtokenizer token = new PGtokenizer(PGtokenizer.removePara(objectStr), ',');
for (int i = 0; i < token.getSize(); i++) {
System.out.println(i + " = " + token.getToken(i) );
}
}

Expected behaviour
0 = 1
1 = 2EC1830300027
2 = 1
3 =
4 =

[davecramer]

Thx for the report

[davecramer]

How is this a problem in the code ? I just tried the same code with StringTokenizer and it only returns 3 tokens.

[chaluvadi286]

To give background - I am trying to map the statement result to a Java Object in a generic way so that we don't need to touch this mapping even if there is a change (addition/modification/deletion) in underlying Type object.

Sample code that I am trying is pasted below. Since PGtokenizer is ignoring token which are null at the end, I am getting ArrayIndexOutofBounds exception.

public TenorObject readSQL(PGobject pgObject) throws Exception {
m_dcfTenor = new TenorObject();
Field[] flds = TenorObject.getPublicFields();

PGtokenizer token = new PGtokenizer(PGtokenizer.removePara(pgObject.getValue()), ',');

for (int i = 0; i < flds.length; i++) {
    if (flds[i].getModifiers() != Modifier.PRIVATE) {//This check is for private variables
	try {
	    if (flds[i].getType().getName().equals("FDate")) {
		flds[i].set(m_dcfTenor, FormatDate.string2FDate(token.getToken(i).replace("\"", ""), "yyyy-MM-dd HH:mm:ss"));
	    } else if (flds[i].getType().getName().equals("java.lang.String")) {
		flds[i].set(m_dcfTenor, token.getToken(i));
	    } else if (flds[i].getType().isInstance(new BigDecimal(0.0))) {
		if (token.getToken(i) != null && !token.getToken(i).equals("")) {
			flds[i].set(m_dcfTenor, new BigDecimal(token.getToken(i)));
		} else {
			flds[i].set(m_dcfTenor, null);
		}
	    }
	} catch (Exception e) {
	    e.printStackTrace();
	}
    }
}	
return m_dcfTenor;

}

If you compare with StringTokenizer, it ignores even in between null values whereas PGTokenizer considers that as value