Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kill DataConverter #2554

merged 2 commits into from Apr 9, 2019

Kill DataConverter #2554

merged 2 commits into from Apr 9, 2019


Copy link

@mikeshardmind mikeshardmind commented Apr 5, 2019

With changes to config, the converter from V2 data is no longer safe for use. (see #2545, #2550, #2551, #2536)for the final nails in the coffin, was already unsafe for use with large datasets)

This can be rewritten at a later date more up to date with newer methods of doing things in Config, but the current version is broken in multiple ways.

  1. It attempts to write each entry individually: This will disconnect some bots currently with a blocked heartbeat. This cannot be fixed by context switching between writes without the potential for data loss.
  2. It has no awareness of the new custom groups, and cannot have that awareness without a large re-design.
  3. The inner workings of it were written hastily, and while they are correct in the transformative process, they are not readily maintainable.

1 & 3 are failings on my part, 2 is a result of changes unforeseen at the time.

This removes the cog and related util, and leaves the door open to anyone wishing to rewrite this up to our current code standards.

End users can still convert data so long as they do so prior to the 3.1 release, but I'd suggest leaving the v2 data behind from here.

tekulvw approved these changes Apr 9, 2019
@tekulvw tekulvw merged commit 136e781 into Cog-Creators:V3/develop Apr 9, 2019
1 check passed
fixator10 added a commit to fixator10/Fixator10-Cogs that referenced this issue Apr 11, 2019
DataConverter does not exists anymore: Cog-Creators/Red-DiscordBot#2554
aikaterna added a commit to aikaterna/Fixator10-Cogs that referenced this issue Jun 4, 2019
* [leveler] some cleanup

fix not handling levelup on disabled messages

* [minecraftdata] new cmd: [p]minecraft server <IP>

* [minecraftdata] fix exception if no players on server

* [minecraftdata] catch query timeout

* [minecraftdata] cleanup server's motd
add cooldown
add typing indicator on data recieveing

* [minecraftdata] fix exception for another type of motd (description)

* [minecraftdata] change some strings

* [minecraftdata] attempt to add server's icon to embed

* [minecraftdata] fix TypeError

TypeError: argument should be a bytes-like object or ASCII string, not 'list'

* [leveler] fix an exception in rank with text mod enabled

* [minecraftdata][generalchannel] use autohelp

* [leveler] ignore member-related commands if member is bot

* [leveler] fix an exception when user is None

* [leveler] add requirements

* [leveler] move abstract bg to profile backgrounds

* [minecraftdata] upload skins to discord instead of use direct link to craftar


* [minecraftdata] fix exception

* [minecraftdata] [?] no attachments in author url

* [minecraftdata] [?] no spaces in URL?

* [leveler] fix an exception in [p]rep

* [leveler] remove duplicate image

* [leveler] fix predicate is recognizing any message as "yes"

* [adminutils] fix error in [p]purge when days<0

* [adminutils] some "restyle"

* [leveler] use bank currency

* [leveler] fix last change

* [leveler] fix last change once again

* [leveler] use only first character of credits name

* [leveler] move [p]badge and [p]role to [p]lvladmin

* [minecraftdata] fix regex

* [leveler] ignore length of message if it has an attachment(s)

* [minecraftdata] make some changes in MCNickname converter

* [minecraftdata] add discord typing while getting data from crafatar

* [leveler] make (almost) everything an coroutine

* [leveler] fix SyntaxError

* [leveler] move badge from lvladmin to lvlset

* [moreutils] add requirement

* [datautils] fix exception in [p]cperms when channel is not provided

* update README

* [translators] "restyle" [p]ytranslate output

* [datautils] escape names

truncate rolenames

* [datautils] change style of [p]roles

* [minecraftdata] add checks for embed permission

* [minecraftdata] support for any type of MOTD

* [minecraftdata] fix TypeError

* [translators] run YandexTranslate methods in executor

* [translators] fix TypeError

* [steamcommunity] add more data to [p]getserver
execute ServerQuerier in executor

* [steamcommunity] move map under servername in [p]getserver

* [steamcommunity] add more executors to prevent blocking of async

* [minecraftdata] refactor some code

* Revert "[minecraftdata] refactor some code"

This reverts commit d7aff5d

* [minecraftdata] refactor some code

* Black code format

* [adminutils] add cooldown for massnick and resetnicks cmds

add examples of emoji commands usage

* [godvilledata] add some info to installation

* [translators] update requirements

* [weather] show state, if provided
use direct string formatting instead of chat_formatting
(as long as it is escapes other formatting)
place does not requires double quotes if has spaces anymore

* [weather] change some formatting

* [weather] change some formatting, again

* [weather] fix AttributeError

* [weather] some reformatting again

* add "code style | black" badge

*more badges for god of badges*

* [translators] rewrite [p]ytranslate to custom API wrapper

* [weather] fix error in DMs

* update

* update README

* remove [p]convertv2
DataConverter does not exists anymore: Cog-Creators/Red-DiscordBot#2554

* [leveler] unused variables cleanup

* [leveler] replace check for owner with redbot.is_owner

owner-specific commands should work now and for co-owners

* [leveler] fix AttributeError

* [leveler] fix [p]mention not working

* fix codefactor issues

* [adminutils] fix error after from latest commit

* [weather] wrap potential requests in executor

fix possible exception from ebd7337

* [leveler] optimize imports

* [leveler] reformat code

* [leveler] reformat code

* [leveler] add more async funcs
remove credits formatting from text truncation

* [leveler] remove notation formatting

* [leveler] rewrite [p]lvlset badge available

* [leveler] fix [p]lvlset badge available when all badges removed

* [leveler] change top output to codeblock instead of single lines

partial copy of cb7487a

* [leveler] fix TypeError from latest commit

* [leveler] fix NameError from latest commit

* [leveler] remove unnecessary formats

* [leveler] fix [p]top
add logging
do not use bare `except:` (this should fix #4)

* [leveler] fix [p]top again

* [leveler] fix None in `[p]top -rep`

* [leveler] some code refactoring

* [leveler] attempt to use BytesIO instead of writing to disk

* [leveler] attempt to use BytesIO instead of writing to disk [2]

* [datautils] better Union usage

* [datautils] better Union usage

* [datautils] fix AttributeError

* [minecraftdata] fix some black formatting failure

* [leveler] order listbadge and listrole

* [leveler] fix order

* [leveler] black formatting

* [leveler] fix lint issue `expression-not-assigned`

* [minecraftdata] fix server ip parse in [p]mc server

* [datautils] do not show webhooks if no webhooks exists in cinfo

* replace all  with  for better visibility

* black formatting

* [leveler] make more keyword-only arguments

* [leveler] update requirements

* [leveler] remove scipy requirement

* [minecraftdata] try-except for ip resolve in [p]mc server

* [minecraftdata] add return

* [moreutils] use rewrite's channel func to get member for `[p]someone`

* update .gitignore

* [weather] add i18n support

* [translators] add i18n support

* [steamcommunity] add i18n support

* [personalroles] add i18n support

* [moreutils] add i18n support

* [minecraftdata] add i18n support

* [generalchannel] add i18n support

* [datautils] add i18n support

* [adminutils] add i18n support

* [weather] add russian translation

* [translators] fix button name

* [translators] update some texts

* [translators] russian translation

* [translators] fix KeyError

* [translators] fix russian translation for new version

* [translators] fix russian translation for new version

* [translators] fix russian translation for new version

* [translators] fix russian translation for new version

* [translators] fix exception with incorrect base64 string

* [translators] fix exception with incorrect base64 string

* [translators] change string again

* [steamcommunity] fix exception output formatting

* [steamcommunity] add russian translation
update translation template

* [steamcommunity] support for i18n in converters

* [personalroles] fix error when server doesn't have any assigned role

* [personalroles] i18n support + russian translation

* [minecraftdata] i18n support update + russian translation

* [moreutils] russian translation

* [generalchannel] russian translation

* [generalchannel] move formatting

* [datautils] add users field for text channels

* [datautils] update translations + russian trunslation

* [adminutils] add some cooldowns to massnicks

* [adminutils] counter for resetnicks

* [adminutils] russian translation

* [adminutils] add format()

* [adminutils] fix kwarg in resetnicks

* [adminutils] better exception handling in ``[p]emoji add`

fix error when image is incorrect

* [adminutils] russian translation

* [adminutils] russian translation update

* black formatting

* update

* update

* [leveler] add `[p]xpban` cmd
add event `leveler_process_exp(message, exp)`

* [datautils] fix exception when role is empty

* [messageslog] new cog

* [messageslog] add info.json

* [messageslog] add some checks
add channel mention

* [messageslog] add exception handling for embeds sending

* update

* fix i18n filenames

* reformat code

* use dict.get instead of str.replace

* [adminutils]reformat code

* update i18n for latest commits

* [leveler] reset cooldown if bot provided

* [minecraftdata] change cooldown pool for [p]mc server

* [leveler] remove cmd cooldown for [p]rep

* [personalroles] add exception handling for HTTPException
reset cooldown if HTTPException occurred

* [generalchannel] reset cooldown if HTTPException occurred

* [personalroles] update ru-RU localization

* [personalroles] add left users cleanup cmd

* Revert "[personalroles] add left users cleanup cmd"

This reverts commit 976c8e1

* [personalroles] display deleted roles as id instead of nothing

* [minecraftdata] add i18n support for one string

* [minecraftdata] update ru-RU localization

* [personalroles] add check for role is exist

make check async

* [steamcommunity] refactor some code

* [weather] update

* [adminutils] prevent editing emoji from another server

* [leveler] fix textonly profile when user has no badges

* Support for Red V3.1 ( 1.0.1) (#7)

* [datautils] update for v3.1

* [messageslog] update for v3.1

* [minecraftdata] update for v3.1

* [personalroles] update for v3.1

* [minecraftdata] black style

* [personalroles] change get_user_info to fetch_user

* [leveler] update for V3.1

* [leveler] fix ValueError

ValueError: embedded null byte

* [leveler] seek 0 position of BytesIO

* [minecraftdata] clear formatting in players list
use server's host and port instead of user's input

* [minecraftdata] refactor code

* [steamcommunity] refactor code

* [steamcommunity] check for incorrect steamid32

* [datautils] add banner field to serverinfo
[steamcommunity] update locales

* [messageslog] makes [p]ignore one cmd, instead of group of commands

* [messageslog] black code formatting

some aliases for main command

* [leveler] update info.json

* [minecraftdata] optimise converter
optimise exception handling

* [smmdata] new cog

* [smmdata] rename ru_RU translation file

* [smmdata] update ru-RU translation

* [smmdata] fix exception when no one is cleared the level

* update

* [smmdata] fixes exception when level doesn't have difficulty yet

* [smmdata] fixes another exception when level doesn't have difficulty yet

* [smmdata] return float instead of None

* [smmdata] refactor code

add icon and level's tag to embed

* [smmdata] do not show footer text, if no tag set

* fix cog unloading

* update .gitignore

* [weather] update version key in info.json

* [minecraftdata] refactor code, fix some nickname issues

* supress PyMissingConstructor warning

* [minecraftdata] add `[p]mc cape` command (getting mojang capes)

do some code refactoring

* [smmdata] add an Game Style field

* [smmdata] add [p]maker cmd

refactor code

* [smmdata] more i18n support

* Revert "[smmdata] more i18n support"

This reverts commit 1e7e4ee

* [smmdata] minor ru-RU translation update

* [translators] kinda refactor [p]googlesay

* [smmdata] add "Uploaded courses" field

* [translators] update translations for [p]googlesay refactor

* [messageslog] fix linting issue

* [leveler] use message_without_command event instead of prefix checking

* [minecraftdata] refactor commands definition

* [reverseimagesearch] new cog

tracemoe is returns 500 for me, needs investigation

* [reverseimagesearch] update info.json

* [reverseimagesearch] implement tracemoe

* [smmdata] fix exception when level was uploaded minutes ago

* [leveler] dispatch `leveler_levelup` event regardless of enabled levelup notifications

* [messageslog] support for ignoring categories

* [leveler] old style with 3.1 support

* [leveler] remove dataconverter

* Fix server images on rank banner and cog unload
@mikeshardmind mikeshardmind deleted the kill-dataconverter branch Dec 26, 2019
@jack1142 jack1142 added this to the 3.1.0 milestone Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
None yet
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants