Filter Not Working

[sebas4055]

Description
I'm building an NFL team stats display using an ESP32 with TFT display. The device fetches game schedule data from ESPN's API (http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule).
The JSON response is approximately 100KB+ and causes ArduinoJson to fail with "InvalidInput" error even with 64KB buffer allocation. JSON filters also fail to work properly - even with proper filter syntax, the parsed document contains 0 events despite the raw JSON containing multiple game events. The goal is to extract team record, next game opponent, date/time, and venue information.

Troubleshooter's report

1. The program uses ArduinoJson 7
2. The issue happens at run time
3. The issue concerns serialization
4. Output is empty
5. JsonDocument::overflowed() returns false

Environment

• Microcontroller: ESP32-DevKitC-32
• Core/Framework: esp32 by Espressif Systems v3.2.1
• IDE: Arduino IDE 2.3.6

Reproduction code

#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

void setup() {
  Serial.begin(115200);
  WiFi.begin("SSID", "PASSWORD");
  while (WiFi.status() != WL_CONNECTED) delay(500);
}

void loop() {
  HTTPClient http;
  http.begin("http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule");
  
  if (http.GET() == 200) {
    // Create filter using .add<JsonObject>() syntax from ArduinoJson Assistant
    JsonDocument filter;
    filter["team"]["recordSummary"] = true;
    
    JsonObject filter_events_0 = filter["events"].add<JsonObject>();
    filter_events_0["date"] = true;
    filter_events_0["week"]["number"] = true;
    
    JsonObject filter_events_0_competitions_0 = filter_events_0["competitions"].add<JsonObject>();
    filter_events_0_competitions_0["status"]["type"]["state"] = true;
    
    JsonObject filter_events_0_competitions_0_competitors_0 = filter_events_0_competitions_0["competitors"].add<JsonObject>();
    filter_events_0_competitions_0_competitors_0["team"]["displayName"] = true;
    
    filter["byeWeek"] = true;
    
    // Parse with filter
    Stream& stream = http.getStream();
    JsonDocument doc;
    
    DeserializationError error = deserializeJson(doc, stream, DeserializationOption::Filter(filter));
    
    Serial.print("Error: ");
    Serial.println(error.c_str());
    Serial.print("Memory used: ");
    Serial.println(doc.memoryUsage());  // Shows 0
    Serial.print("Events found: ");
    Serial.println(doc["events"].size());  // Expected: 17, Actual: 0
  }
  
  http.end();
  delay(60000);
}

Remarks
The raw JSON payload can be retrieved successfully and is valid JSON
Other smaller API endpoints parse correctly with the same code
Increasing buffer size beyond 65KB causes memory allocation failures
This appears to be a common issue with large sports API responses on ESP32
Looking for suggestions on streaming JSON parsing or alternative approaches for handling large API responses with limited memory

full code below:
#include <TFT_eSPI.h>
#include "mode_1.h"
#include "mode_2.h"

#include <WiFi.h>
#include <HTTPClient.h>
#include <ArduinoJson.h>

struct GameData {
// Live game data
String homeTeam;
String awayTeam;
int homeScore;
int awayScore;
String quarter;
String timeRemaining;
String gameStatus;
bool isGameLive;

// Next game data
String nextOpponent;
String nextGameDay;
String nextGameMonth;
String nextGameTime;
bool isHome;

// Season stats
int wins;
int losses;
int byeWeek;
float pointsPerGame;
float yardsPerGame;
float passYardsPerGame;
float rushYardsPerGame;
float pointsAllowedPerGame;
float yardsAllowedPerGame;
float passYardsAllowedPerGame;
float rushYardsAllowedPerGame;

bool isBroncosGame;
};

GameData currentGame;
unsigned long lastUpdate = 0;
const unsigned long updateInterval = 30000; // 30 seconds

const char* ssid = "wifi";
const char* password = "pswd";
const char* scheduleUrl = "http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/schedule";
const char* statsUrl = "http://site.api.espn.com/apis/site/v2/sports/football/nfl/teams/den/statistics";

TFT_eSPI tft = TFT_eSPI();

#define BTTN_PIN 13

int displayMode = 0;
int lastState = HIGH;
int currentState;

void connectWiFi() {

Serial.print("Connecting to WiFi");
WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) {
delay(500);
Serial.print(".");
}

Serial.println("\nWiFi Connected!");
Serial.print("IP: ");
Serial.println(WiFi.localIP());
}

void setup() {
Serial.begin(115200);
pinMode(BTTN_PIN, INPUT_PULLUP);
tft.init();
tft.setRotation(1);
tft.setSwapBytes(true);
tft.fillScreen(TFT_BLACK);
connectWiFi();
}

void loop() {
currentState = digitalRead(BTTN_PIN);

if (currentState == LOW && lastState == HIGH) {
displayMode = (displayMode + 1) % 3;

if (displayMode == 1) {
  // Mode 1: General stats and next game
  tft.pushImage(0, 0, 320, 240, mode_1);
  Serial.println("Mode 1: General Stats");
  fetchGeneralData();
  //debugPrintRawJson();
  //displayGeneralMode();
} else if (displayMode == 2) {
  // Mode 2: Live game
  tft.pushImage(0, 0, 320, 240, mode_2);
  Serial.println("Mode 2: Live Game");
  //fetchLiveGameData();
  displayLiveMode();
} else {
  tft.fillScreen(TFT_BLACK);
  Serial.println("Mode 0: Off");
}

Serial.println("button");

}

lastState = currentState;
}
void fetchGeneralData() {
if (WiFi.status() != WL_CONNECTED) {
Serial.println("WiFi disconnected");
return;
}

Serial.println("\n========== FETCHING DATA ==========");

HTTPClient http;
http.begin(scheduleUrl);
http.setTimeout(20000);

int httpCode = http.GET();
Serial.printf("HTTP Response Code: %d\n", httpCode);

if (httpCode != 200) {
Serial.println("❌ HTTP error");
http.end();
return;
}

// --- 1) Build FILTER document (from ArduinoJson Assistant) ---
// IMPORTANT: give it a capacity (e.g. 2048), NOT JsonDocument filter;
JsonDocument filter;

filter["team"]["recordSummary"] = true;
JsonObject filter_events_0 = filter["events"].add();
filter_events_0["name"] = true;
filter_events_0["shortName"] = true;
filter_events_0["date"] = true;
filter_events_0["seasonType"]["abbreviation"] = true;
filter_events_0["week"]["number"] = true;

JsonObject filter_events_0_competitions_0 = filter_events_0["competitions"].add();
filter_events_0_competitions_0["date"] = true;
filter_events_0_competitions_0["venue"]["address"]["city"] = true;

JsonObject filter_events_0_competitions_0_competitors_0 = filter_events_0_competitions_0["competitors"].add();
filter_events_0_competitions_0_competitors_0["order"] = true;
filter_events_0_competitions_0_competitors_0["team"]["displayName"] = true;
filter_events_0_competitions_0_competitors_0["score"]["value"] = true;
filter_events_0_competitions_0_competitors_0["record"][0]["displayValue"] = true;

JsonObject filter_events_0_competitions_0_status = filter_events_0_competitions_0["status"].to();
filter_events_0_competitions_0_status["clock"] = true;
filter_events_0_competitions_0_status["displayClock"] = true;
filter_events_0_competitions_0_status["period"] = true;
filter_events_0_competitions_0_status["type"]["state"] = true;
filter["byeWeek"] = true;

// (Optional) debug: show what the filter looks like
Serial.println("Filter doc:");
serializeJsonPretty(filter, Serial);
Serial.println();

// --- 2) Deserialize with filter from the HTTP stream ---
WiFiClient &input = http.getStream();

JsonDocument doc; // result after filtering (much smaller than raw JSON)

DeserializationError error = deserializeJson(
doc, input,
DeserializationOption::Filter(filter),
DeserializationOption::NestingLimit(30)); // bump nesting limit a bit

if (error) {
Serial.print("deserializeJson() failed: ");
Serial.println(error.c_str());
http.end();
return;
}

// Debug: check memory usage + preview
Serial.print("overflowed: ");
Serial.println(doc.overflowed());
Serial.print("memoryUsage: ");
Serial.println(doc.memoryUsage());
Serial.println("Filtered doc preview:");
serializeJsonPretty(doc, Serial);
Serial.println();

Serial.println("✓ JSON parsed successfully!");

// --- 3) team.recordSummary -> wins/losses ---
const char* team_recordSummary = doc["team"]["recordSummary"]; // "9-2"
if (team_recordSummary) {
int w = 0, l = 0;
if (sscanf(team_recordSummary, "%d-%d", &w, &l) == 2) {
currentGame.wins = w;
currentGame.losses = l;
Serial.printf("✓ Parsed record: %d-%d\n", w, l);
} else {
Serial.printf("⚠ Could not parse recordSummary: %s\n", team_recordSummary);
}
} else {
Serial.println("⚠ recordSummary missing");
}

// --- 4) Access events[] (you can later add your next-game logic here) ---
JsonArray events = doc["events"].as();
if (events.isNull() || events.size() == 0) {
Serial.println("⚠ No events in schedule");
http.end();
return;
}

Serial.printf("Events in filtered doc: %u\n", events.size());

// Example: print basic info for first event
JsonObject e0 = events[0];
const char* e0_date = e0["date"] | "";
const char* e0_name = e0["name"] | "";
const char* e0_shortName = e0["shortName"] | "";
int e0_week = e0["week"]["number"] | 0;

Serial.printf("Week %d: %s (%s) on %s\n",
e0_week, e0_name, e0_shortName, e0_date);

JsonObject comp0 = e0["competitions"][0];
const char* city = comp0["venue"]["address"]["city"] | "";
Serial.printf(" Venue city: %s\n", city);

JsonArray competitors = comp0["competitors"].as();
for (JsonObject c : competitors) {
int order = c["order"] | -1;
const char* teamName = c["team"]["displayName"] | "";
int score = c["score"]["value"] | -1;

Serial.printf("  [%d] %s score=%d\n", order, teamName, score);

}

int byeWeek = doc["byeWeek"] | -1;
if (byeWeek > 0) {
currentGame.byeWeek = byeWeek;
Serial.printf("Bye week: %d\n", byeWeek);
}

http.end();
}

void displayGeneralMode() {
Serial.println("\n========== DISPLAY: GENERAL MODE ==========");
Serial.printf("Record: %d-%d\n", currentGame.wins, currentGame.losses);
Serial.printf("Next Opponent: %s (%s)\n",
currentGame.nextOpponent.c_str(),
currentGame.isHome ? "HOME" : "AWAY");
Serial.printf("Points Per Game: %.1f\n", currentGame.pointsPerGame);
Serial.printf("Yards Per Game: %.1f\n", currentGame.yardsPerGame);
Serial.printf("Pass Yards Per Game: %.1f\n", currentGame.passYardsPerGame);
Serial.printf("Rush Yards Per Game: %.1f\n", currentGame.rushYardsPerGame);
Serial.println("===========================================\n");
}

void displayLiveMode() {
Serial.println("\n========== DISPLAY: LIVE GAME MODE ==========");

if (currentGame.isGameLive) {
Serial.printf("%s vs %s\n", currentGame.homeTeam.c_str(), currentGame.awayTeam.c_str());
Serial.printf("Score: %d - %d\n", currentGame.homeScore, currentGame.awayScore);
Serial.printf("Quarter: %s\n", currentGame.quarter.c_str());
Serial.printf("Time Remaining: %s\n", currentGame.timeRemaining.c_str());
} else {
Serial.println("No live game currently");
}

Serial.println("=============================================\n");
}

/*
// Display functions - you'll implement these based on your UI
void displayGeneralMode() {
// Draw the general stats on screen
tft.setTextColor(TFT_WHITE, TFT_NAVY);
tft.setTextSize(2);

// Example:
tft.setCursor(10, 10);
tft.printf("Record: %d-%d", currentGame.wins, currentGame.losses);

tft.setCursor(10, 40);
tft.printf("Next: vs %s", currentGame.nextOpponent.c_str());

tft.setCursor(10, 70);
tft.printf("PPG: %.1f", currentGame.pointsPerGame);

// Add more display code here
}

void displayLiveMode() {
// Draw the live game info on screen
if (currentGame.isGameLive) {
tft.setTextColor(TFT_WHITE, TFT_NAVY);
tft.setTextSize(3);

tft.setCursor(50, 80);
tft.printf("%d - %d", currentGame.homeScore, currentGame.awayScore);

tft.setTextSize(2);
tft.setCursor(80, 120);
tft.printf("Q%s %s", currentGame.quarter.c_str(), currentGame.timeRemaining.c_str());

} else {
tft.setTextSize(2);
tft.setCursor(60, 100);
tft.print("No live game");
}
}*/

[bblanchon]

Hi @sebas4055,

causes ArduinoJson to fail with "InvalidInput" error

This radically differs from the answers you gave to the Troubleshooter.
Please give it another chance and try the most relevant options.

Best regards,
Benoit

[sebas4055]

Hi Benoit,
Thanks for replying and sorry for the confusion in my initial report!

Let me try and explain the situation better:

When I deserializeJson(doc, stream) without a filter it fails with InvalidInput error because the raw JSON response is ~100KB+, which exceeds available memory

When I try it with the filter, deserializeJson(doc, stream, DeserializationOption::Filter(filter)) succeeds (no error reported)
However, the resulting document is completely empty, it outputs:

doc.memoryUsage() returns 0
doc["events"].size() returns 0
doc["team"]["recordSummary"] is null
All fields are missing despite being included in the filter
and also
doc.overflowed() returns false

I've tried multiple versions of the filter,
First I used the one generated using the ArduinoJson Assistant-generated filter. This one used .add() syntax for arrays
Example: JsonObject filter_events_0 = filter["events"].add();
When printed with serializeJsonPretty(filter, Serial), the filter structure looks correct and matches the actual JSON schema and seemed to work on the assistant website but still gave me an empty doc.

Then I tried an alternative array filter syntax. I change to: filter["events"][0].to() notation (based on some examples suggesting this creates array templates)
Also tried filter["events"].createNestedObject()
All variations produce the same empty document

Environment:

ESP32-DevKitC (ESP8266 module)
ArduinoJson 7.4.0
Also tried Stream& stream = http.getStream() and http.getString() with same results

The core issue is filtering appears to work, but produces a completely empty document even though the filter structure is correct.

Is there a known issue with filtered deserialization from HTTP streams in v7.4, or am I missing something about how filters should be structured for streaming deserialization?
Happy to provide the full filter structure or raw JSON sample if that helps!

Thanks,
Sebastian

[bblanchon]

it fails with InvalidInput error because the raw JSON response is ~100KB+, which exceeds available memory

I stopped reading there.
InvalidInput means the input is invalid, not too big.

Can you please give the ArduinoJson Troubleshooter a try?