-
|
Describe the bug Error while crawling /tmp/es: Mapping is incorrect: please set stored: true on field [file.filename]. Job Settings ---
name: "test2"
fs:
url: "/tmp/es"
update_rate: "1m"
excludes:
- "*/~*"
json_support: false
filename_as_id: false
add_filesize: true
remove_deleted: true
add_as_inner_object: false
store_source: false
index_content: true
attributes_support: false
raw_metadata: false
xml_support: false
index_folders: true
lang_detect: false
continue_on_error: false
ocr:
language: "eng"
enabled: true
pdf_strategy: "ocr_and_text"
follow_symlinks: false
elasticsearch:
nodes:
- url: "http://10.8.25.140:9200"
bulk_size: 100
flush_interval: "5s"
byte_size: "10mb"
ssl_verification: true
Logs Expected behavior A clear and concise description of what you expected to happen. Versions: |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 8 replies
-
|
Hey You should check the mapping as the logs say. |
Beta Was this translation helpful? Give feedback.
-
|
I'm using the default configuration |
Beta Was this translation helpful? Give feedback.
-
|
"file": { |
Beta Was this translation helpful? Give feedback.
-
|
Please format your code with the |
Beta Was this translation helpful? Give feedback.
-
_settings_folder.json{
"settings": {
"analysis": {
"analyzer": {
"fscrawler_path": {
"tokenizer": "fscrawler_path"
}
},
"tokenizer": {
"fscrawler_path": {
"type": "path_hierarchy"
}
}
}
},
"mappings": {
"properties" : {
"real" : {
"type" : "keyword",
"store" : true
},
"root" : {
"type" : "keyword",
"store" : true
},
"virtual" : {
"type" : "keyword",
"store" : true
}
}
}
}_settings.json{
"settings": {
"number_of_shards": 1,
"index.mapping.total_fields.limit": 2000,
"analysis": {
"analyzer": {
"fscrawler_path": {
"tokenizer": "fscrawler_path"
}
},
"tokenizer": {
"fscrawler_path": {
"type": "path_hierarchy"
}
}
}
},
"mappings": {
"dynamic_templates": [
{
"raw_as_text": {
"path_match": "meta.raw.*",
"mapping": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
],
"properties": {
"attachment": {
"type": "binary",
"doc_values": false
},
"attributes": {
"properties": {
"group": {
"type": "keyword"
},
"owner": {
"type": "keyword"
}
}
},
"content": {
"type": "text"
},
"file": {
"properties": {
"content_type": {
"type": "keyword"
},
"filename": {
"type": "keyword",
"store": true
},
"extension": {
"type": "keyword"
},
"filesize": {
"type": "long"
},
"indexed_chars": {
"type": "long"
},
"indexing_date": {
"type": "date",
"format": "dateOptionalTime"
},
"created": {
"type": "date",
"format": "dateOptionalTime"
},
"last_modified": {
"type": "date",
"format": "dateOptionalTime"
},
"last_accessed": {
"type": "date",
"format": "dateOptionalTime"
},
"checksum": {
"type": "keyword"
},
"url": {
"type": "keyword",
"index": false
}
}
},
"meta": {
"properties": {
"author": {
"type": "text"
},
"date": {
"type": "date",
"format": "dateOptionalTime"
},
"keywords": {
"type": "text"
},
"title": {
"type": "text"
},
"language": {
"type": "keyword"
},
"format": {
"type": "text"
},
"identifier": {
"type": "text"
},
"contributor": {
"type": "text"
},
"coverage": {
"type": "text"
},
"modifier": {
"type": "text"
},
"creator_tool": {
"type": "keyword"
},
"publisher": {
"type": "text"
},
"relation": {
"type": "text"
},
"rights": {
"type": "text"
},
"source": {
"type": "text"
},
"type": {
"type": "text"
},
"description": {
"type": "text"
},
"created": {
"type": "date",
"format": "dateOptionalTime"
},
"print_date": {
"type": "date",
"format": "dateOptionalTime"
},
"metadata_date": {
"type": "date",
"format": "dateOptionalTime"
},
"latitude": {
"type": "text"
},
"longitude": {
"type": "text"
},
"altitude": {
"type": "text"
},
"rating": {
"type": "byte"
},
"comments": {
"type": "text"
}
}
},
"path": {
"properties": {
"real": {
"type": "keyword",
"fields": {
"tree": {
"type": "text",
"analyzer": "fscrawler_path",
"fielddata": true
},
"fulltext": {
"type": "text"
}
}
},
"root": {
"type": "keyword"
},
"virtual": {
"type": "keyword",
"fields": {
"tree": {
"type": "text",
"analyzer": "fscrawler_path",
"fielddata": true
},
"fulltext": {
"type": "text"
}
}
}
}
}
}
}
}
|
Beta Was this translation helpful? Give feedback.
-
|
Please reply, this problem is very urgent, I want to know how to skip this error |
Beta Was this translation helpful? Give feedback.
-
|
I don't have time ATM to answer as I'm working on the Elastic Community Conference which is happening next week. But to move forward, I'm pretty sure that you shared here the configuration files which are distributed with FSCrawler and not the actual mapping which you can get with: Please check those ones and if needed share the result here. |
Beta Was this translation helpful? Give feedback.
-
|
Use the index with the test keyword is OK, use other not OK, very magical |
Beta Was this translation helpful? Give feedback.
-
|
I found a problem. If I create the corresponding es index manually, it will happen |
Beta Was this translation helpful? Give feedback.
-
|
I had the issue but could not find anything wrong with the mappings. However, I remembered that I renamed my index during my attempts to connect FSCralwer to ElasticSearch. After I deleted my indexes, the problem disappeared and FSCrawler was able to complete the indexing process. |
Beta Was this translation helpful? Give feedback.
Hey
You should check the mapping as the logs say.