Skip to content

Fail to create the ElasticSearch indices #18625

Closed
Bug
#27258
Closed
Fail to create the ElasticSearch indices#18625
Bug
#27258
Labels
bugSomething isn't working

Description

@rbernardes
rbernardes
opened on Jun 8, 2022

Steps to reproduce the problem

  1. Clean mastodon install with openjdk and elasticsearch
  2. Create some posts
  3. Try to run "RAILS_ENV=production bin/tootctl search deploy" to create indices with data

Expected behaviour

A message or indicator that indexes have been created

Actual behaviour

You can't set the item's total value to less than the current progress. (ProgressBar::InvalidProgressError)

Specifications

Mastodon 3.53
Ubuntu Server 20.04.4
openjdk version "17.0.3" 2022-04-19
elasticsearch 7.17.4
Postgres 14.3-1

mastodon@grendel:~/live$ RAILS_ENV=production bin/tootctl search deploy
warning: 299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."
warning: 299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."
warning: 299 Elasticsearch-7.17.4-79878662c54c886ae89206c685d9f1051a9d6411 "Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-minimal-setup.html to enable security."
/home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/progress.rb:76:in `total=': You can't set the item's total value to less than the current progress. (ProgressBar::InvalidProgressError)ocs/s)
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:178:in `block in update_progress'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/output.rb:43:in `with_refresh'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:177:in `update_progress'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:101:in `total='
	from /home/mastodon/live/lib/mastodon/search_cli.rb:67:in `deploy'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
	from /home/mastodon/live/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
	from bin/tootctl:8:in `<main>'
mastodon@grendel:~/live$

Activity

  1. rbernardes
    added
    bugSomething isn't working
    on Jun 8, 2022

  2. gunchleoc commented on Oct 15, 2022

    @gunchleoc
    gunchleoc
    on Oct 15, 2022
    Contributor

    I can reproduce this. I got rid of the security error messages by adding the lines below to /etc/elasticsearch/elasticsearch.yml. Note that the security role that I picked has wide powers, because I just wanted to get this running at all - a more restrictive role might be more appropriate.

    discovery.type: single-node

xpack.security.enabled: true

xpack.security.authc:
  anonymous:
    username: anonymous_user
    roles: superuser
    authz_exception: true

    This did not fix the progressbar error though.

    I'm on a production server running Mastodon v3.5.3. OS is Debian 11.

  3. gunchleoc commented on Oct 21, 2022

    @gunchleoc
    gunchleoc
    on Oct 21, 2022
    Contributor

    After running the server for a few days, I can no longer reproduce this problem. So, it must be related to an initial state somehow.

  4. oliverjkb commented on Nov 2, 2022

    @oliverjkb
    oliverjkb
    on Nov 2, 2022

    I'm not a ruby dev, but what I found was, that
    /home/mastodon/live/lib/mastodon/search_cli.rb:67:in deploy' seems to point to the right direction.

    If I change progress.total = indices.sum { |index| importers[index].estimate! } to just assigning an arbitrary number like
    progress.total = 2000 then the command runs 'just fine' and indices get created.
    Seems like the calculation of indices fails and hence is nil (or whatever the ruby equivalent is), which causes progressbar so throw an error.

    This somehow makes sense, as the initial progress (which would be zero, I guess) would be higher than the calculated total of nil/undefined/whatever.

    Maybe that's of any help.

  5. rmacd commented on Nov 5, 2022

    @rmacd
    rmacd
    on Nov 5, 2022

    Replicated here as well

    Mastodon 4.0.0rc1
    Ruby 3.0.4p208
    PostgreSQL 14.5
    Redis 6.2.7

    $ RAILS_ENV=production bin/tootctl search deploy

    /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/progress.rb:76:in `total=': You can't set the item's total value to less than the current progress. (ProgressBar::InvalidProgressError)
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:178:in `block in update_progress'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/output.rb:43:in `with_refresh'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:177:in `update_progress'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:101:in `total='
        from /opt/mastodon/lib/mastodon/search_cli.rb:67:in `deploy'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
        from bin/tootctl:8:in `<main>'

    ES:

    {
  "name" : "example",
  "cluster_name" : "example",
  "cluster_uuid" : "z3LNO3tdTN2oDlKcVSYgOw",
  "version" : {
    "number" : "7.17.7",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "78dcaaa8cee33438b91eca7f5c7f56a70fec9e80",
    "build_date" : "2022-10-17T15:29:54.167373105Z",
    "build_snapshot" : false,
    "lucene_version" : "8.11.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

    Using authentication on ES; user has access to ONLY indices under its own namespace (defaults to REDIS_NAMESPACE).

    pcap indicates it's creating the indices no problem, no 404s being returned during this process:

    POST /masteh28_chewy_specifications/_search HTTP/1.1
Authorization: Basic aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj1kUXc0dzlXZ1hjUQ
x-elastic-client-meta: es=7.13.3,rb=3.0.4,t=7.13.3,fd=1.9.3,hc=2.8.3
User-Agent: elasticsearch-ruby/7.13.3 (RUBY_VERSION: 3.0.4; linux x86_64; Faraday v1.9.3)
Content-Type: application/json
Accept: */*
Accept-Encoding: gzip,deflate
Date: Sat, 05 Nov 2022 17:05:00 GMT
Content-Length: 70
Host: 172.17.0.1:9200

{"size":1,"query":{"bool":{"filter":{"ids":{"values":["accounts"]}}}}}

HTTP/1.1 200 OK
X-elastic-product: Elasticsearch
content-type: application/json; charset=UTF-8
content-encoding: gzip
content-length: 743
<data>

    Can confirm that hacky fix by @oliverjkb works here; within my mastodon web docker image, just ran:

    sed -E 's/indices.sum.+/2000/g' -i lib/mastodon/search_cli.rb

  6. meepmeep commented on Nov 7, 2022

    @meepmeep
    meepmeep
    on Nov 7, 2022

    Tried
    sed -E 's/indices.sum.+/2000/g' -i lib/mastodon/search_cli.rb

    But I still have an error but I get a part of a progress bar:

    /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/progress.rb:76:in `total=': You can't set the item's total value to less than the current progress. (ProgressBar::InvalidProgressError)--=---=---=---=---=-|  ETA: ??:??:?? (0 docs/s)
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:178:in `block in update_progress'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/output.rb:43:in `with_refresh'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:177:in `update_progress'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/ruby-progressbar-1.11.0/lib/ruby-progressbar/base.rb:101:in `total='
        from /opt/mastodon/lib/mastodon/search_cli.rb:67:in `deploy'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:116:in `invoke'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:243:in `block in subcommand'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/command.rb:27:in `run'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/invocation.rb:127:in `invoke_command'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor.rb:392:in `dispatch'
        from /opt/mastodon/vendor/bundle/ruby/3.0.0/gems/thor-1.2.1/lib/thor/base.rb:485:in `start'
        from bin/tootctl:8:in `<main>'

  7. rmacd commented on Nov 7, 2022

    @rmacd
    rmacd
    on Nov 7, 2022

    Are you exporting an already extant, massive database? In which case I suspect s/2000/2000000/g - hacky as it might be - might do the trick?

  8. meepmeep commented on Nov 7, 2022

    @meepmeep
    meepmeep
    on Nov 7, 2022

    brand new install with 1 user, 2 posts , so no, not a massive db ;)

  9. rmacd commented on Nov 7, 2022

    @rmacd
    rmacd
    on Nov 7, 2022

    Odd; can you confirm the versions you're on? (preferences > admin > "software" box)

  10. meepmeep commented on Nov 7, 2022

    @meepmeep
    meepmeep
    on Nov 7, 2022

    Software
    Mastodon 3.5.3
    Ruby 3.0.3p157
    PostgreSQL 14.5
    Redis 7.0.5

    (create with docker-compose, from a git pull this morning)

  11. meepmeep commented on Nov 8, 2022

    @meepmeep
    meepmeep
    on Nov 8, 2022

    Seems to work today... Don't know why !

  12. nodomain commented on Nov 9, 2022

    @nodomain
    nodomain
    on Nov 9, 2022

    Same here when deploying to AWS OpenSearch 1.3 with activated compatibility mode

    PUT /_cluster/settings
{
  "persistent": {
    "compatibility.override_main_response_version": true
  }
}

    image

  13. navjotjsingh commented on Nov 13, 2022

    @navjotjsingh
    navjotjsingh
    on Nov 13, 2022

    Facing the same error and @oliverjkb 's fix didn't work. Maybe it will work after a few days. Is there a fix coming?

  14. rjbrown99 commented on Nov 21, 2022

    @rjbrown99
    rjbrown99
    on Nov 21, 2022

    Ran into this today, the sed command above worked to resolve it. I'm also a brand new Mastodon site with 2 users (both me) and 0 posts.

  15. oculos commented on Nov 21, 2022

    @oculos
    oculos
    on Nov 21, 2022

    Yesterday it didn't work. Today it did...

  16. rmacd commented on Nov 21, 2022

    @rmacd
    rmacd
    on Nov 21, 2022

    to be honest I'd be tempted to just remove this feature rather than try fix; progress bar offers no inherent value vs potentially causing buggy deploys.

  17. dariusk
    added a commit that references this issue on Jan 17, 2023

    Dockerfile update (#1270)

    d7e2a5c

  18. GiantFrog commented on Jul 26, 2023

    @GiantFrog
    GiantFrog
    on Jul 26, 2023

    For what it's worth, I'm also getting a ProgressBar::InvalidProgressError on a new server without any posts yet. I'll just wait for a while and try again.

  19. jgillich
    added a commit that references this issue on Oct 3, 2023

    Fix importer returning negative row estimates

    6570625

  20. jgillich commented on Oct 3, 2023

    @jgillich
    jgillich
    on Oct 3, 2023
    Contributor

    I've submitted a fix for this issue, please test if anyone's still getting this: #27258

  21. ClearlyClaire
    closed this as completedin #27258on Oct 3, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    Bug

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions