Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] MappingIT testMapperDynamicIndexSetting failing #109455

Closed
mark-vieira opened this issue Jun 6, 2024 · 3 comments
Closed

[CI] MappingIT testMapperDynamicIndexSetting failing #109455

mark-vieira opened this issue Jun 6, 2024 · 3 comments
Assignees
@martijnvg
Labels
low-risk An open issue or test failure that is a low risk to future releases :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine >test-failure Triaged test failures from CI

Comments

@mark-vieira
Copy link
Contributor

This looks to be a new test added in #109301

Build scan:
https://gradle-enterprise.elastic.co/s/hoxn5f3dd54hk/tests/:qa:rolling-upgrade:v7.13.4%23oneThirdUpgradedTest/org.elasticsearch.upgrades.MappingIT/testMapperDynamicIndexSetting

Reproduction line:

./gradlew ':qa:rolling-upgrade:v7.13.4#oneThirdUpgradedTest' -Dtests.class="org.elasticsearch.upgrades.MappingIT" -Dtests.method="testMapperDynamicIndexSetting" -Dtests.seed=272DB9A7614B8058 -Dtests.bwc=true -Dtests.locale=es-CO -Dtests.timezone=Asia/Baghdad -Druntime.java=22

Applicable branches:
7.17

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.upgrades.MappingIT#testMapperDynamicIndexSetting

Failure excerpt:

java.lang.AssertionError: timed out waiting for green state for index [my-index] cluster state [{
  "cluster_name" : "v7.13.4",
  "cluster_uuid" : "jdJyCc9nRbK52Z3qrblQrA",
  "version" : 387,
  "state_uuid" : "ugOc_dkoSqu0CeTWhRysTw",
  "master_node" : "5SaXoi7LQv-rMGdQWoEcsg",
  "blocks" : {
    "indices" : {
      "closed_index_created_on_old" : {
        "4" : {
          "description" : "index closed",
          "retryable" : false,
          "levels" : [
            "read",
            "write"
          ]
        }
      },
      "index_old_7130499" : {
        "4" : {
          "description" : "index closed",
          "retryable" : false,
          "levels" : [
            "read",
            "write"
          ]
        }
      },
      "closed_index_replica_allocation" : {
        "4" : {
          "description" : "index closed",
          "retryable" : false,
          "levels" : [
            "read",
            "write"
          ]
        }
      }
    }
  },
  "nodes" : {
    "DN1nowVMRZeeiWTNpNoxxQ" : {
      "name" : "v7.13.4-2",
      "ephemeral_id" : "bC1AVhnJSeaCCm5qK9AI4g",
      "transport_address" : "127.0.0.1:42289",
      "attributes" : {
        "testattr" : "test",
        "ml.machine_memory" : "126604300288",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "536870912",
        "transform.node" : "true"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    },
    "5SaXoi7LQv-rMGdQWoEcsg" : {
      "name" : "v7.13.4-1",
      "ephemeral_id" : "qDGsIvPzSc2cfALfpe0psA",
      "transport_address" : "127.0.0.1:34839",
      "attributes" : {
        "testattr" : "test",
        "ml.machine_memory" : "126604300288",
        "ml.max_open_jobs" : "512",
        "xpack.installed" : "true",
        "ml.max_jvm_size" : "536870912",
        "transform.node" : "true"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    },
    "vz9cOsqBRGSQfSLU8TOO0w" : {
      "name" : "v7.13.4-0",
      "ephemeral_id" : "dHnz-_AlSn6QKeLPjJyrVA",
      "transport_address" : "127.0.0.1:43783",
      "attributes" : {
        "ml.machine_memory" : "126604300288",
        "upgraded" : "true",
        "xpack.installed" : "true",
        "transform.node" : "true",
        "testattr" : "test",
        "ml.max_open_jobs" : "512",
        "ml.max_jvm_size" : "536870912"
      },
      "roles" : [
        "data",
        "data_cold",
        "data_content",
        "data_frozen",
        "data_hot",
        "data_warm",
        "ingest",
        "master",
        "ml",
        "remote_cluster_client",
        "transform"
      ]
    }
  },
  "metadata" : {
    "cluster_uuid" : "jdJyCc9nRbK52Z3qrblQrA",
    "cluster_uuid_committed" : true,
    "cluster_coordination" : {
      "term" : 1,
      "last_committed_config" : [
        "DN1nowVMRZeeiWTNpNoxxQ",
        "vz9cOsqBRGSQfSLU8TOO0w",
        "5SaXoi7LQv-rMGdQWoEcsg"
      ],
      "last_accepted_config" : [
        "DN1nowVMRZeeiWTNpNoxxQ",
        "vz9cOsqBRGSQfSLU8TOO0w",
        "5SaXoi7LQv-rMGdQWoEcsg"
      ],
      "voting_config_exclusions" : [ ]
    },
    "templates" : {
      ".ml-stats" : {
        "order" : 0,
        "version" : 7130499,
        "index_patterns" : [
          ".ml-stats-*"
        ],
        "settings" : {
          "index" : {
            "lifecycle" : {
              "name" : "ml-size-based-ilm-policy",
              "rollover_alias" : ".ml-stats-write"
            },
            "hidden" : "true",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1"
          }
        },
        "mappings" : {
          "_doc" : {
            "_meta" : {
              "version" : "7.13.4"
            },
            "dynamic" : false,
            "properties" : {
              "skipped_docs_count" : {
                "type" : "long"
              },
              "validation_loss" : {
                "properties" : {
                  "fold_values" : {
                    "properties" : {
                      "fold" : {
                        "type" : "integer"
                      },
                      "values" : {
                        "type" : "double"
                      }
                    }
                  },
                  "loss_type" : {
                    "type" : "keyword"
                  }
                }
              },
              "cache_miss_count" : {
                "type" : "long"
              },
              "timing_stats" : {
                "properties" : {
                  "iteration_time" : {
                    "type" : "long"
                  },
                  "elapsed_time" : {
                    "type" : "long"
                  }
                }
              },
              "failure_count" : {
                "type" : "long"
              },
              "model_id" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "keyword"
              },
              "training_docs_count" : {
                "type" : "long"
              },
              "inference_count" : {
                "type" : "long"
              },
              "job_id" : {
                "type" : "keyword"
              },
              "missing_all_fields_count" : {
                "type" : "long"
              },
              "peak_usage_bytes" : {
                "type" : "long"
              },
              "iteration" : {
                "type" : "integer"
              },
              "hyperparameters" : {
                "properties" : {
                  "max_attempts_to_add_tree" : {
                    "type" : "integer"
                  },
                  "downsample_factor" : {
                    "type" : "double"
                  },
                  "eta_growth_rate_per_tree" : {
                    "type" : "double"
                  },
                  "soft_tree_depth_tolerance" : {
                    "type" : "double"
                  },
                  "max_trees" : {
                    "type" : "integer"
                  },
                  "lambda" : {
                    "type" : "double"
                  },
                  "max_optimization_rounds_per_hyperparameter" : {
                    "type" : "integer"
                  },
                  "eta" : {
                    "type" : "double"
                  },
                  "soft_tree_depth_limit" : {
                    "type" : "double"
                  },
                  "alpha" : {
                    "type" : "double"
                  },
                  "class_assignment_objective" : {
                    "type" : "keyword"
                  },
                  "feature_bag_fraction" : {
                    "type" : "double"
                  },
                  "num_splits_per_feature" : {
                    "type" : "integer"
                  },
                  "gamma" : {
                    "type" : "double"
                  },
                  "num_folds" : {
                    "type" : "integer"
                  }
                }
              },
              "parameters" : {
                "properties" : {
                  "compute_feature_influence" : {
                    "type" : "boolean"
                  },
                  "feature_influence_threshold" : {
                    "type" : "double"
                  },
                  "outlier_fraction" : {
                    "type" : "double"
                  },
                  "method" : {
                    "type" : "keyword"
                  },
                  "standardization_enabled" : {
                    "type" : "boolean"
                  },
                  "n_neighbors" : {
                    "type" : "integer"
                  }
                }
              },
              "test_docs_count" : {
                "type" : "long"
              },
              "node_id" : {
                "type" : "keyword"
              },
              "timestamp" : {
                "type" : "date"
              }
            }
          }
        },
        "aliases" : { }
      },
      ".monitoring-alerts-7" : {
        "order" : 0,
        "version" : 7120099,
        "index_patterns" : [
          ".monitoring-alerts-7"
        ],
        "settings" : {
          "index" : {
            "format" : "7",
            "codec" : "best_compression",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : {
          "_doc" : {
            "dynamic" : false,
            "properties" : {
              "metadata" : {
                "properties" : {
                  "severity" : {
                    "type" : "short"
                  },
                  "cluster_uuid" : {
                    "type" : "keyword"
                  },
                  "watch" : {
                    "type" : "keyword"
                  },
                  "link" : {
                    "type" : "keyword"
                  },
                  "type" : {
                    "type" : "keyword"
                  },
                  "version" : {
                    "type" : "keyword"
                  }
                }
              },
              "update_timestamp" : {
                "type" : "date"
              },
              "prefix" : {
                "type" : "text"
              },
              "message" : {
                "type" : "text"
              },
              "suffix" : {
                "type" : "text"
              },
              "resolved_timestamp" : {
                "type" : "date"
              },
              "timestamp" : {
                "type" : "date"
              }
            }
          }
        },
        "aliases" : { }
      },
      ".monitoring-logstash" : {
        "order" : 0,
        "version" : 7120099,
        "index_patterns" : [
          ".monitoring-logstash-7-*"
        ],
        "settings" : {
          "index" : {
            "format" : "7",
            "codec" : "best_compression",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : {
          "_doc" : {
            "dynamic" : false,
            "properties" : {
              "interval_ms" : {
                "type" : "long"
              },
              "logstash_state" : {
                "properties" : {
                  "pipeline" : {
                    "properties" : {
                      "batch_size" : {
                        "type" : "integer"
                      },
                      "format" : {
                        "type" : "keyword"
                      },
                      "id" : {
                        "type" : "keyword"
                      },
                      "ephemeral_id" : {
                        "type" : "keyword"
                      },
                      "workers" : {
                        "type" : "short"
                      },
                      "version" : {
                        "type" : "keyword"
                      },
                      "representation" : {
                        "enabled" : false
                      },
                      "hash" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "http_address" : {
                    "type" : "keyword"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "host" : {
                    "type" : "keyword"
                  },
                  "ephemeral_id" : {
                    "type" : "keyword"
                  },
                  "uuid" : {
                    "type" : "keyword"
                  },
                  "version" : {
                    "type" : "keyword"
                  },
                  "snapshot" : {
                    "type" : "boolean"
                  },
                  "status" : {
                    "type" : "keyword"
                  }
                }
              },
              "logstash_stats" : {
                "type" : "object",
                "properties" : {
                  "jvm" : {
                    "properties" : {
                      "mem" : {
                        "properties" : {
                          "heap_used_percent" : {
                            "type" : "long"
                          },
                          "heap_max_in_bytes" : {
                            "type" : "long"
                          },
                          "heap_used_in_bytes" : {
                            "type" : "long"
                          }
                        }
                      },
                      "uptime_in_millis" : {
                        "type" : "long"
                      },
                      "gc" : {
                        "properties" : {
                          "collectors" : {
                            "properties" : {
                              "young" : {
                                "properties" : {
                                  "collection_count" : {
                                    "type" : "long"
                                  },
                                  "collection_time_in_millis" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "old" : {
                                "properties" : {
                                  "collection_count" : {
                                    "type" : "long"
                                  },
                                  "collection_time_in_millis" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "logstash" : {
                    "properties" : {
                      "pipeline" : {
                        "properties" : {
                          "batch_size" : {
                            "type" : "long"
                          },
                          "workers" : {
                            "type" : "short"
                          }
                        }
                      },
                      "http_address" : {
                        "type" : "keyword"
                      },
                      "name" : {
                        "type" : "keyword"
                      },
                      "host" : {
                        "type" : "keyword"
                      },
                      "ephemeral_id" : {
                        "type" : "keyword"
                      },
                      "uuid" : {
                        "type" : "keyword"
                      },
                      "version" : {
                        "type" : "keyword"
                      },
                      "snapshot" : {
                        "type" : "boolean"
                      },
                      "status" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "process" : {
                    "properties" : {
                      "open_file_descriptors" : {
                        "type" : "long"
                      },
                      "max_file_descriptors" : {
                        "type" : "long"
                      },
                      "cpu" : {
                        "properties" : {
                          "percent" : {
                            "type" : "long"
                          }
                        }
                      }
                    }
                  },
                  "pipelines" : {
                    "type" : "nested",
                    "properties" : {
                      "vertices" : {
                        "type" : "nested",
                        "properties" : {
                          "events_out" : {
                            "type" : "long"
                          },
                          "pipeline_ephemeral_id" : {
                            "type" : "keyword"
                          },
                          "events_in" : {
                            "type" : "long"
                          },
                          "double_gauges" : {
                            "type" : "nested",
                            "properties" : {
                              "name" : {
                                "type" : "keyword"
                              },
                              "value" : {
                                "type" : "double"
                              }
                            }
                          },
                          "queue_push_duration_in_millis" : {
                            "type" : "long"
                          },
                          "duration_in_millis" : {
                            "type" : "long"
                          },
                          "id" : {
                            "type" : "keyword"
                          },
                          "long_counters" : {
                            "type" : "nested",
                            "properties" : {
                              "name" : {
                                "type" : "keyword"
                              },
                              "value" : {
                                "type" : "long"
                              }
                            }
                          }
                        }
                      },
                      "id" : {
                        "type" : "keyword"
                      },
                      "ephemeral_id" : {
                        "type" : "keyword"
                      },
                      "hash" : {
                        "type" : "keyword"
                      },
                      "events" : {
                        "properties" : {
                          "filtered" : {
                            "type" : "long"
                          },
                          "in" : {
                            "type" : "long"
                          },
                          "queue_push_duration_in_millis" : {
                            "type" : "long"
                          },
                          "duration_in_millis" : {
                            "type" : "long"
                          },
                          "out" : {
                            "type" : "long"
                          }
                        }
                      },
                      "queue" : {
                        "properties" : {
                          "events_count" : {
                            "type" : "long"
                          },
                          "max_queue_size_in_bytes" : {
                            "type" : "long"
                          },
                          "queue_size_in_bytes" : {
                            "type" : "long"
                          },
                          "type" : {
                            "type" : "keyword"
                          }
                        }
                      },
                      "reloads" : {
                        "properties" : {
                          "failures" : {
                            "type" : "long"
                          },
                          "successes" : {
                            "type" : "long"
                          }
                        }
                      }
                    }
                  },
                  "batch_size" : {
                    "type" : "integer"
                  },
                  "os" : {
                    "properties" : {
                      "cpu" : {
                        "properties" : {
                          "load_average" : {
                            "properties" : {
                              "5m" : {
                                "type" : "half_float"
                              },
                              "15m" : {
                                "type" : "half_float"
                              },
                              "1m" : {
                                "type" : "half_float"
                              }
                            }
                          }
                        }
                      },
                      "cgroup" : {
                        "properties" : {
                          "cpu" : {
                            "properties" : {
                              "stat" : {
                                "properties" : {
                                  "number_of_elapsed_periods" : {
                                    "type" : "long"
                                  },
                                  "number_of_times_throttled" : {
                                    "type" : "long"
                                  },
                                  "time_throttled_nanos" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "control_group" : {
                                "type" : "keyword"
                              }
                            }
                          },
                          "cpuacct" : {
                            "properties" : {
                              "control_group" : {
                                "type" : "keyword"
                              },
                              "usage_nanos" : {
                                "type" : "long"
                              }
                            }
                          }
                        }
                      }
                    }
                  },
                  "workers" : {
                    "type" : "short"
                  },
                  "events" : {
                    "properties" : {
                      "filtered" : {
                        "type" : "long"
                      },
                      "in" : {
                        "type" : "long"
                      },
                      "duration_in_millis" : {
                        "type" : "long"
                      },
                      "out" : {
                        "type" : "long"
                      }
                    }
                  },
                  "queue" : {
                    "properties" : {
                      "events_count" : {
                        "type" : "long"
                      },
                      "type" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "timestamp" : {
                    "type" : "date"
                  },
                  "reloads" : {
                    "properties" : {
                      "failures" : {
                        "type" : "long"
                      },
                      "successes" : {
                        "type" : "long"
                      }
                    }
                  }
                }
              },
              "cluster_uuid" : {
                "type" : "keyword"
              },
              "source_node" : {
                "properties" : {
                  "transport_address" : {
                    "type" : "keyword"
                  },
                  "ip" : {
                    "type" : "keyword"
                  },
                  "host" : {
                    "type" : "keyword"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "uuid" : {
                    "type" : "keyword"
                  },
                  "timestamp" : {
                    "format" : "date_time",
                    "type" : "date"
                  }
                }
              },
              "type" : {
                "type" : "keyword"
              },
              "timestamp" : {
                "format" : "date_time",
                "type" : "date"
              }
            }
          }
        },
        "aliases" : { }
      },
      ".transform-notifications-000002" : {
        "order" : 0,
        "version" : 7130499,
        "index_patterns" : [
          ".transform-notifications-*"
        ],
        "settings" : {
          "index" : {
            "hidden" : "true",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1"
          }
        },
        "mappings" : {
          "_doc" : {
            "_meta" : {
              "version" : "7.13.4"
            },
            "dynamic" : "false",
            "properties" : {
              "level" : {
                "type" : "keyword"
              },
              "transform_id" : {
                "type" : "keyword"
              },
              "node_name" : {
                "type" : "keyword"
              },
              "message" : {
                "type" : "text",
                "fields" : {
                  "raw" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              },
              "timestamp" : {
                "type" : "date"
              }
            }
          }
        },
        "aliases" : {
          ".transform-notifications-read" : {
            "is_hidden" : true
          }
        }
      },
      ".monitoring-beats" : {
        "order" : 0,
        "version" : 7120099,
        "index_patterns" : [
          ".monitoring-beats-7-*"
        ],
        "settings" : {
          "index" : {
            "format" : "7",
            "codec" : "best_compression",
            "number_of_shards" : "1",
            "auto_expand_replicas" : "0-1",
            "number_of_replicas" : "0"
          }
        },
        "mappings" : {
          "_doc" : {
            "dynamic" : false,
            "properties" : {
              "interval_ms" : {
                "type" : "long"
              },
              "cluster_uuid" : {
                "type" : "keyword"
              },
              "source_node" : {
                "properties" : {
                  "ip" : {
                    "type" : "keyword"
                  },
                  "transport_address" : {
                    "type" : "keyword"
                  },
                  "host" : {
                    "type" : "keyword"
                  },
                  "name" : {
                    "type" : "keyword"
                  },
                  "uuid" : {
                    "type" : "keyword"
                  }
                }
              },
              "type" : {
                "type" : "keyword"
              },
              "beats_stats" : {
                "properties" : {
                  "beat" : {
                    "properties" : {
                      "host" : {
                        "type" : "keyword"
                      },
                      "name" : {
                        "type" : "keyword"
                      },
                      "type" : {
                        "type" : "keyword"
                      },
                      "uuid" : {
                        "type" : "keyword"
                      },
                      "version" : {
                        "type" : "keyword"
                      }
                    }
                  },
                  "metrics" : {
                    "properties" : {
                      "system" : {
                        "properties" : {
                          "load" : {
                            "properties" : {
                              "1" : {
                                "type" : "double"
                              },
                              "15" : {
                                "type" : "double"
                              },
                              "5" : {
                                "type" : "double"
                              },
                              "norm" : {
                                "properties" : {
                                  "1" : {
                                    "type" : "double"
                                  },
                                  "15" : {
                                    "type" : "double"
                                  },
                                  "5" : {
                                    "type" : "double"
                                  }
                                }
                              }
                            }
                          }
                        }
                      },
                      "beat" : {
                        "properties" : {
                          "handles" : {
                            "properties" : {
                              "limit" : {
                                "properties" : {
                                  "hard" : {
                                    "type" : "long"
                                  },
                                  "soft" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "open" : {
                                "type" : "long"
                              }
                            }
                          },
                          "cpu" : {
                            "properties" : {
                              "total" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "value" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "system" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "user" : {
                                "properties" : {
                                  "ticks" : {
                                    "type" : "long"
                                  },
                                  "time" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "memstats" : {
                            "properties" : {
                              "rss" : {
                                "type" : "long"
                              },
                              "gc_next" : {
                                "type" : "long"
                              },
                              "memory_alloc" : {
                                "type" : "long"
                              },
                              "memory_total" : {
                                "type" : "long"
                              }
                            }
                          },
                          "cgroup" : {
                            "properties" : {
                              "memory" : {
                                "properties" : {
                                  "mem" : {
                                    "properties" : {
                                      "usage" : {
                                        "properties" : {
                                          "bytes" : {
                                            "type" : "long"
                                          }
                                        }
                                      },
                                      "limit" : {
                                        "properties" : {
                                          "bytes" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              },
                              "cpu" : {
                                "properties" : {
                                  "cfs" : {
                                    "properties" : {
                                      "period" : {
                                        "properties" : {
                                          "us" : {
                                            "type" : "long"
                                          }
                                        }
                                      },
                                      "quota" : {
                                        "properties" : {
                                          "us" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "stats" : {
                                    "properties" : {
                                      "periods" : {
                                        "type" : "long"
                                      },
                                      "throttled" : {
                                        "properties" : {
                                          "ns" : {
                                            "type" : "long"
                                          },
                                          "periods" : {
                                            "type" : "long"
                                          }
                                        }
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              },
                              "cpuacct" : {
                                "properties" : {
                                  "total" : {
                                    "properties" : {
                                      "ns" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "id" : {
                                    "type" : "keyword"
                                  }
                                }
                              }
                            }
                          },
                          "info" : {
                            "properties" : {
                              "ephemeral_id" : {
                                "type" : "keyword"
                              },
                              "uptime" : {
                                "properties" : {
                                  "ms" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          }
                        }
                      },
                      "libbeat" : {
                        "properties" : {
                          "output" : {
                            "properties" : {
                              "read" : {
                                "properties" : {
                                  "bytes" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "type" : {
                                "type" : "keyword"
                              },
                              "write" : {
                                "properties" : {
                                  "bytes" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "events" : {
                                "properties" : {
                                  "batches" : {
                                    "type" : "long"
                                  },
                                  "duplicates" : {
                                    "type" : "long"
                                  },
                                  "total" : {
                                    "type" : "long"
                                  },
                                  "dropped" : {
                                    "type" : "long"
                                  },
                                  "toomany" : {
                                    "type" : "long"
                                  },
                                  "active" : {
                                    "type" : "long"
                                  },
                                  "failed" : {
                                    "type" : "long"
                                  },
                                  "acked" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          },
                          "pipeline" : {
                            "properties" : {
                              "clients" : {
                                "type" : "long"
                              },
                              "events" : {
                                "properties" : {
                                  "total" : {
                                    "type" : "long"
                                  },
                                  "filtered" : {
                                    "type" : "long"
                                  },
                                  "dropped" : {
                                    "type" : "long"
                                  },
                                  "active" : {
                                    "type" : "long"
                                  },
                                  "failed" : {
                                    "type" : "long"
                                  },
                                  "published" : {
                                    "type" : "long"
                                  },
                                  "retry" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "queue" : {
                                "properties" : {
                                  "acked" : {
                                    "type" : "long"
                                  }
                                }
                              }
                            }
                          },
                          "config" : {
                            "properties" : {
                              "module" : {
                                "properties" : {
                                  "running" : {
                                    "type" : "long"
                                  },
                                  "stops" : {
                                    "type" : "long"
                                  },
                                  "starts" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "reloads" : {
                                "type" : "long"
                              }
                            }
                          }
                        }
                      },
                      "apm-server" : {
                        "properties" : {
                          "server" : {
                            "properties" : {
                              "request" : {
                                "properties" : {
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "response" : {
                                "properties" : {
                                  "valid" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "accepted" : {
                                        "type" : "long"
                                      },
                                      "ok" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "count" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "properties" : {
                                      "ratelimit" : {
                                        "type" : "long"
                                      },
                                      "toolarge" : {
                                        "type" : "long"
                                      },
                                      "internal" : {
                                        "type" : "long"
                                      },
                                      "method" : {
                                        "type" : "long"
                                      },
                                      "unauthorized" : {
                                        "type" : "long"
                                      },
                                      "forbidden" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "closed" : {
                                        "type" : "long"
                                      },
                                      "decode" : {
                                        "type" : "long"
                                      },
                                      "queue" : {
                                        "type" : "long"
                                      },
                                      "validate" : {
                                        "type" : "long"
                                      },
                                      "concurrency" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              },
                              "concurrent" : {
                                "properties" : {
                                  "wait" : {
                                    "properties" : {
                                      "ms" : {
                                        "type" : "long"
                                      }
                                    }
                                  }
                                }
                              }
                            }
                          },
                          "acm" : {
                            "properties" : {
                              "request" : {
                                "properties" : {
                                  "count" : {
                                    "type" : "long"
                                  }
                                }
                              },
                              "response" : {
                                "properties" : {
                                  "valid" : {
                                    "properties" : {
                                      "notmodified" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "accepted" : {
                                        "type" : "long"
                                      },
                                      "ok" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "request" : {
                                    "properties" : {
                                      "count" : {
                                        "type" : "long"
                                      }
                                    }
                                  },
                                  "count" : {
                                    "type" : "long"
                                  },
                                  "unset" : {
                                    "type" : "long"
                                  },
                                  "errors" : {
                                    "properties" : {
                                      "ratelimit" : {
                                        "type" : "long"
                                      },
                                      "internal" : {
                                        "type" : "long"
                                      },
                                      "toolarge" : {
                                        "type" : "long"
                                      },
                                      "method" : {
                                        "type" : "long"
                                      },
                                      "unavailable" : {
                                        "type" : "long"
                                      },
                                      "forbidden" : {
                                        "type" : "long"
                                      },
                                      "count" : {
                                        "type" : "long"
                                      },
                                      "decode" : {
                                        "type" : "long"
                                      },
                                      "unauthorized" : {
                                        "type" : "long"
                                      },
                                      "notfound" : {
                                        "type" : "long"
          ...
@mark-vieira mark-vieira added :StorageEngine/Mapping The storage related side of mappings >test-failure Triaged test failures from CI labels Jun 6, 2024
mark-vieira added a commit that referenced this issue Jun 6, 2024
@mark-vieira
AwaitsFix: #109455
60c36c0
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@elasticsearchmachine elasticsearchmachine added the needs:risk Requires assignment of a risk label (low, medium, blocker) label Jun 6, 2024
@martijnvg martijnvg self-assigned this Jun 7, 2024
@martijnvg
Copy link
Member

Looks like a test issue. During the rolling upgrade the cluster tries the allocate a replica, but fails 5 times. After which no retry is happening. Even when the upgrade is completed and the primary shard is allocated:

"my-index" : {
            "shards" : {
              "0" : [
                {
                  "state" : "STARTED",
                  "primary" : true,
                  "node" : "vz9cOsqBRGSQfSLU8TOO0w",
                  "relocating_node" : null,
                  "shard" : 0,
                  "index" : "my-index",
                  "allocation_id" : {
                    "id" : "N-QFY6eRRQiYoLuTcdzPqg"
                  }
                },
                {
                  "state" : "UNASSIGNED",
                  "primary" : false,
                  "node" : null,
                  "relocating_node" : null,
                  "shard" : 0,
                  "index" : "my-index",
                  "recovery_source" : {
                    "type" : "PEER"
                  },
                  "unassigned_info" : {
                    "reason" : "ALLOCATION_FAILED",
                    "at" : "2024-06-06T20:29:55.535Z",
                    "failed_attempts" : 5,
                    "failed_nodes" : [
                      "DN1nowVMRZeeiWTNpNoxxQ",
                      "5SaXoi7LQv-rMGdQWoEcsg"
                    ],
                    "delayed" : false,
                    "details" : "failed shard on node [DN1nowVMRZeeiWTNpNoxxQ]: failed to create index, failure IllegalArgumentException[Setting index.mapper.dynamic was removed after version 6.0.0]",
                    "allocation_status" : "no_attempt"
                  }
                }
              ]
            }
          },

@martijnvg martijnvg added low-risk An open issue or test failure that is a low risk to future releases and removed needs:risk Requires assignment of a risk label (low, medium, blocker) labels Jun 7, 2024
martijnvg added a commit to martijnvg/elasticsearch that referenced this issue Jun 7, 2024
@martijnvg
Address MappingIT#testMapperDynamicIndexSetting() test failure
e9bc684 
This is a 7.17 rolling upgrade test.
See the mentioned issue and comment in the test.

Closes elastic#109455
@martijnvg martijnvg mentioned this issue Jun 7, 2024
Merged
martijnvg added a commit that referenced this issue Jun 10, 2024
@martijnvg
Address MappingIT#testMapperDynamicIndexSetting() test failure (#109479)
9021686 
This is a 7.17 rolling upgrade test.

During upgrade health may be yellow, because due to shard relation on old node the shard can't be allocated due to the bug.
Also ensure each node in the cluster has a copy.

Closes #109455
@martijnvg
Copy link
Member

This should be fixed now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@martijnvg martijnvg

Labels
low-risk An open issue or test failure that is a low risk to future releases :StorageEngine/Mapping The storage related side of mappings Team:StorageEngine >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@martijnvg @mark-vieira @elasticsearchmachine