18.2.3 Ingest Attachment:文件处理器插件
sudo bin/elasticsearch-plugin install ingest-attachment
name: "gupao_fs"
fs:
url: "/www/mingyi_file_search_system/fscrawler-es7-2.9/filesystem"
update_rate: "1m"
excludes:
- "*/~*"
json_support: false
filename_as_id: false
add_filesize: true
remove_deleted: true
add_as_inner_object: false
store_source: false
index_content: true
attributes_support: false
raw_metadata: false
xml_support: false
index_folders: true
lang_detect: false
continue_on_error: false
ocr:
language: "eng"
enabled: true
pdf_strategy: "ocr_and_text"
follow_symlinks: false
elasticsearch:
nodes:
- url: "http://172.20.0.14:29190"
bulk_size: 100
flush_interval: "5s"
byte_size: "10mb"
ssl_verification: true
username: elastic
password: changeme
./bin/fscrawler my_crawler_settings.yaml
## 1.初始化
bin/fscrawler --config_dir ./test_03 gupao_fs
创建配置文件
## 2.启动
bin/fscrawler --config_dir ./test_03 gupao_fs
## 3.重启
bin/fscrawler --config_dir ./test_03 gupao_fs restart
## 4.rest方式启动
bin/fscrawler --config_dir ./test gupao_fs --loop 0 --rest
如果端口冲突,记得先配置一下端口。
## 5.rest方式启动+上传文件
echo "This is my text" > test.txt
curl -F "file=@test.txt" "http://127.0.0.1:8080/fscrawler/_upload"
Python 工程代码详见:Chapter18_MingyiFSSearch.zip