Rework whitelist and ops support #1182

Stealthii · 2021-12-11T07:00:04Z

This PR implements a few notable changes to the handling of ops and whitelist configurations, including:

Support for a preconfigured file (via OPS_FILE and WHITELIST_FILE)
Extension of UUID support to OPS and better handling of mixed configurations
Additional handling of txt/json generation in the vanilla 1.7.5 -> 1.7.6 jump (support for usernames & uuids in all server binaries)

Documentation has been updated to clarify the configuration options, and draw attention to the ENFORCE_WHITELIST variable (with a warning being added to the logs when whitelist is enabled but not enforced).

Detailed changes and rationale can be seen in the expanded commit messages.

This commit adds support for providing the JSON file used for ops and whitelist definitions. This is provided via the OPS_JSON and WHITELIST_JSON environment variables, and similar to OPS and WHITELIST they are copied on first run and only override the existing files if OVERRIDE_OPS and OVERRIDE_WHITELIST are set to true. If both forms of definition are provided, the JSON is processed first, with additional processing occuring with the user/uuid CSV variables. This ensures consistent behaviour with hiw options are currently used, and allows for override configuration to be provided at runtime or definition (for example a default whitelist/ops configuration, with additionals added for an event)

Minecraft 1.7.6 added support for UUIDs via the JSON configuration file format for whitelist and ops. To handle 1.7.5 and earlier, we perform slightly different logic. If Minecraft 1.7.5 or earlier: - Delete the correct files for OVERRIDE_OPS/OVERRIDE_WHITELIST - Update ops.txt and white-list.txt with users provided via variables - Error on use of JSON environtment variables As part of this change, a fix to handle existing configuration for both JSON and text-based configurations was added. If existing files are found, we uniquely merge with them.

OPS_JSON and WHITELIST_JSON have been switched to OPS_FILE and WHITELIST_FILE, to allow for their use as text files for Minecraft 1.7.5 and below. The JSON check has been removed for now.

The parser has been rewritten to individually work on usernames/UUIDs passed in the OPS and WHITELIST environment variables. If a username or UUID could not be looked up, we log a warning about the error. No failure state is created and the user is skipped. We no longer use the legacy .txt conversion for Minecraft >= 1.7.6, we now update the ops.json or whitelist.json automatically. For both old txt and new json formats we ensure uniqueness. As part of this change, support for UUIDs in OPS has been added.

As both the csv and file based methods now contain identical code, this logic has been moved into processor functions.

Documentation has been updated to draw attention to the requirement of ENFORCE_WHITELIST to actually enforce the configured whitelist. A server log message has been added to warn if the whitelist functionality is enabled but not enforced. The duplicate setting of 'whitelist' and 'white-list' in server.properties has been removed, leaving only 'white-list' being applied. I believe this was an error introduced with the addition of whitelist functionality in 111883e, that was subsequently fixed in 32cb5f4.

Stealthii · 2021-12-11T07:20:59Z

Testing Ops configuration.

Fresh Ops configuration

Vars: -e 'OPS=Stealthii,Koyleuk' -e 'OVERRIDE_OPS=TRUE'

1.17.1 (ops.json):

[
  {
    "uuid": "89117fd7-1b96-43dd-86c5-9716311b0f03",
    "name": "Koyleuk",
    "level": 4,
    "bypassesPlayerLimit": false
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii",
    "level": 4,
    "bypassesPlayerLimit": false
  }
]

1.7.6 (ops.json):

[
  {
    "uuid": "89117fd7-1b96-43dd-86c5-9716311b0f03",
    "name": "Koyleuk",
    "level": 4
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii",
    "level": 4
  }
]

1.7.5 (ops.txt):

Koyleuk
Stealthii

Update Ops configuration

Vars: -e 'OPS=Stealthii,clairebones' -e 'OVERRIDE_OPS=FALSE'

1.17.1 (ops.json):

[
  {
    "uuid": "89117fd7-1b96-43dd-86c5-9716311b0f03",
    "name": "Koyleuk",
    "level": 4,
    "bypassesPlayerLimit": false
  },
  {
    "uuid": "da641452-04ac-4b7a-a510-20680209760b",
    "name": "clairebones",
    "level": 4,
    "bypassesPlayerLimit": false
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii",
    "level": 4,
    "bypassesPlayerLimit": false
  }
]

1.7.6 (ops.json):

[
  {
    "uuid": "89117fd7-1b96-43dd-86c5-9716311b0f03",
    "name": "Koyleuk",
    "level": 4
  },
  {
    "uuid": "da641452-04ac-4b7a-a510-20680209760b",
    "name": "clairebones",
    "level": 4
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii",
    "level": 4
  }
]

1.7.5 (ops.txt):

clairebones
koyleuk
stealthii

Testing 1.7.5 server with pre-defined `OPS_FILE` and `OPS` addition

Vars: -e 'OPS_FILE=/data/myops.txt' -e 'OPS=Swooooon'

$ cat /data/myops.txt
clairebones
notauserlol
Stealthii

Log:

[init] Recreating ops.txt file at server startup
[init] Copying ops.txt from /data/myops.txt
[init] Updating ops

Output (ops.txt)

clairebones
swooooon
stealthii
notauserlol

Testing UUID and username mix

Vars: -e 'OPS=eb03ad21-ac26-46fa-b102-833acf3f0aa1,Swooooon' -e 'OVERRIDE_OPS=TRUE'

1.7.5 log:

[init] Recreating ops.txt file at server startup
[init] Updating ops

1.7.5 ops.txt:

Stealthii
Swooooon

1.7.6 log

[init] Recreating ops.json file at server startup
[init] Updating ops

1.7.6 ops.json:

[
  {
    "uuid": "42bbffc7-cde1-4872-a58b-037e75a48c3f",
    "name": "Swooooon",
    "level": 4
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii",
    "level": 4
  }
]

Stealthii · 2021-12-11T07:44:04Z

Testing mixed whitelist configuration (without enforcement)
Vars: -e 'WHITELIST=eb03ad21-ac26-46fa-b102-833acf3f0aa1,Swooooon' -e 'OVERRIDE_WHITELIST=TRUE'

[init] Enabling whitelist functionality
[init] Setting white-list to 'true' in /data/server.properties
[init] WARNING: whitelist enabled but not enforced. Set ENFORCE_WHITELIST=TRUE or update 'enforce-whitelist' in server.properties to enforce the whitelist.
[init] Recreating whitelist.json file at server startup
[init] Updating whitelist

whitelist.json example:

[
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii"
  },
  {
    "uuid": "42bbffc7-cde1-4872-a58b-037e75a48c3f",
    "name": "Swooooon"
  }
]

white-list.txt example

Stealthii
Swooooon

Testing with remote whitelist file, and one launch override
Vars: -e 'WHITELIST_FILE=https://pepsi.dog/minecraft/whitelist.json' -e WHITELIST='Dinnerbone' -e 'OVERRIDE_WHITELIST=TRUE'

Logs:

[init] Recreating whitelist.json file at server startup
[init] Downloading whitelist.json from https://pepsi.dog/minecraft/whitelist.json
[init] Updating whitelist

Result (whitelist.json)

[
  {
    "uuid": "02bbc126-accc-4969-bb5f-4d9d8d3bbf57",
    "name": "TMorrow97"
  },
  {
    "uuid": "0632f2e4-ea2c-4073-a643-582af3c405e1",
    "name": "Zakkkkkkk"
  },
  {
    "uuid": "0e7d7b2f-239c-4b14-b513-7bd1983d756e",
    "name": "Vindernator"
  },
  {
    "uuid": "42bbffc7-cde1-4872-a58b-037e75a48c3f",
    "name": "Swooooon"
  },
  {
    "uuid": "61699b2e-d327-4a01-9f1e-0ea8c3f06bc6",
    "name": "Dinnerbone"
  },
  {
    "uuid": "72e246ad-ecd1-4037-82bd-03c2274514cd",
    "name": "Fixxation"
  },
  {
    "uuid": "89117fd7-1b96-43dd-86c5-9716311b0f03",
    "name": "Koyleuk"
  },
  {
    "uuid": "90912008-d23e-4658-a8bd-d7c8ea739b74",
    "name": "corryhewitt"
  },
  {
    "uuid": "940fe915-f71e-423a-a11b-a79d1ec7afd6",
    "name": "HandsBlix"
  },
  {
    "uuid": "bc67e53a-8905-4bd8-baa5-63bbd358fcdb",
    "name": "xstepperz"
  },
  {
    "uuid": "da641452-04ac-4b7a-a510-20680209760b",
    "name": "clairebones"
  },
  {
    "uuid": "eb03ad21-ac26-46fa-b102-833acf3f0aa1",
    "name": "Stealthii"
  },
  {
    "uuid": "f70f7ceb-6123-4e55-a204-8a0533de447a",
    "name": "Agstepperz"
  },
  {
    "uuid": "fd0d9de4-ab1d-4376-b210-4bb3ba7aebf1",
    "name": "randomoctomusic"
  }
]

This significantly speeds up processing for large csv lists.

itzg

Refactorings look great. I did a quick test with OPS=itzg and WHITELIST=itzg,user2. ops.json looked right but whitelist.json ended up with itzg listed with ops structure:

[ { "uuid": "5cddfd26-fc86-4981-b52e-c42bb10bfdef", "name": "itzg", "level": 4 }, { "uuid": "5cddfd26-fc86-4981-b52e-c42bb10bfdef", "name": "itzg" }, { "uuid": "redacted", "name": "user2" } ]

The playerDataList variable was being re-used, and adding to whitelist after Ops processing.

Stealthii · 2021-12-11T21:56:08Z

@itzg good spot. Although not harmful (seems that option is ignored), it was re-use of the playerListData variable, so any entries destined for ops.json got added to whitelist.json too.

Making that local seems to have fixed the issue:
Vars: -e 'OPS=Dinnerbone' -e 'WHITELIST=Agstepperz,Fixxation,Dinnerbone'

$ cat ops.json
[
  {
    "uuid": "61699b2e-d327-4a01-9f1e-0ea8c3f06bc6",
    "name": "Dinnerbone",
    "level": 4
  }
]

$ cat whitelist.json
[
  {
    "uuid": "f70f7ceb-6123-4e55-a204-8a0533de447a",
    "name": "Agstepperz"
  },
  {
    "uuid": "72e246ad-ecd1-4037-82bd-03c2274514cd",
    "name": "Fixxation"
  },
  {
    "uuid": "61699b2e-d327-4a01-9f1e-0ea8c3f06bc6",
    "name": "Dinnerbone"
  }
]

itzg

Adjustment looks good. Thanks!

itzg · 2021-12-11T22:47:16Z

This is now merged to all the variants, tagged, and pushed

Stealthii · 2021-12-12T00:40:23Z

Appreciate it!

michaelilyin · 2022-01-30T08:45:27Z

@Stealthii it there any way now to add player to the whitelist or ops list without call to playerdb API?

Stealthii · 2022-01-30T16:04:11Z

You can still pass UUIDs directly with OPS or WHITELIST, which won't need converted for 1.7.5 or later, or use OPS_FILE or WHITELIST_FILE with a predetermined list. If your server can't make the call out to playerdb, I'd recommend looking up the player IDs yourself and adding these to the config. This way there's no ambiguity with username swapping.

…

On Sun, 30 Jan 2022, 08:45 Michael Ilyin, ***@***.***> wrote: @Stealthii <https://github.com/Stealthii> it there any way now to add player to the whitelist or ops list without call to playerdb API? — Reply to this email directly, view it on GitHub <#1182 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHT4CENU6SLE3HYQYVCCCLUYT3DFANCNFSM5J2VVEDQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

Stealthii added 8 commits December 11, 2021 04:56

Clear ops/whitelist json before processing if override set

4c6f66e

Implement file support for pre-1.7.6 releases

f16c33c

OPS_JSON and WHITELIST_JSON have been switched to OPS_FILE and WHITELIST_FILE, to allow for their use as text files for Minecraft 1.7.5 and below. The JSON check has been removed for now.

Condense ops/whitelist processing into functions

abbd037

As both the csv and file based methods now contain identical code, this logic has been moved into processor functions.

Documentation changes

e2b6618

Skip usernames & UUIDs already present in config

831085b

This significantly speeds up processing for large csv lists.

Stealthii force-pushed the feature/user-json branch from 3e05807 to 831085b Compare December 11, 2021 09:30

itzg reviewed Dec 11, 2021

View reviewed changes

Fix issue with level being added to whitelist.json

392cd70

The playerDataList variable was being re-used, and adding to whitelist after Ops processing.

itzg approved these changes Dec 11, 2021

View reviewed changes

itzg merged commit e9326db into itzg:master Dec 11, 2021

Stealthii deleted the feature/user-json branch December 12, 2021 00:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Rework whitelist and ops support #1182

Rework whitelist and ops support #1182

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

itzg left a comment

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

itzg left a comment

Uh oh!

itzg commented Dec 11, 2021

Uh oh!

Stealthii commented Dec 12, 2021

Uh oh!

michaelilyin commented Jan 30, 2022

Uh oh!

Stealthii commented Jan 30, 2022 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Rework whitelist and ops support #1182

Rework whitelist and ops support #1182

Uh oh!

Conversation

Stealthii commented Dec 11, 2021

Uh oh!

Stealthii commented Dec 11, 2021

Fresh Ops configuration

Update Ops configuration

Testing 1.7.5 server with pre-defined OPS_FILE and OPS addition

Testing UUID and username mix

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

itzg left a comment

Choose a reason for hiding this comment

Uh oh!

Stealthii commented Dec 11, 2021

Uh oh!

itzg left a comment

Choose a reason for hiding this comment

Uh oh!

itzg commented Dec 11, 2021

Uh oh!

Stealthii commented Dec 12, 2021

Uh oh!

michaelilyin commented Jan 30, 2022

Uh oh!

Stealthii commented Jan 30, 2022 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Testing 1.7.5 server with pre-defined `OPS_FILE` and `OPS` addition