python manage.py migrate not working #57

vigneshshettyin · 2023-11-04T06:34:57Z

Describe the bug
python manage.py migrate not working

To Reproduce
Create a new django app try connecting to Clickhouse then run migrate command

Expected behavior
It should create all default tables without any error

Logs
$ python manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, polls, sessions
Running migrations:
Traceback (most recent call last):
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\dbapi\cursor.py", line 111, in execute
response = execute(
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_backend\driver\client.py", line 53, in execute
rv = self.process_ordinary_query(
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\client.py", line 571, in process_ordinary_query
return self.receive_result(with_column_types=with_column_types,
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\client.py", line 204, in receive_result
return result.get_result()
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\result.py", line 50, in get_result
for packet in self.packet_generator:
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\client.py", line 220, in packet_generator
packet = self.receive_packet()
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\client.py", line 237, in receive_packet
raise packet.exception
clickhouse_driver.errors.ServerException: Code: 57.
DB::Exception: Table django_local.django_migrations already exists. Stack trace:

DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cdd11b7 in /usr/bin/clickhouse
DB::Exception::Exception<String&, String, String>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity::type, std::type_identity::type>, String&, String&&, String&&) @ 0x0000000006c9791e in /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_deleteDB::DDLGuard>&) @ 0x00000000125e309f in /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000125d9ddf in /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute() @ 0x00000000125e725e in /usr/bin/clickhouse
DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000012be45f5 in /usr/bin/clickhouse
DB::executeQuery(String const&, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum) @ 0x0000000012bde2fe in /usr/bin/clickhouse
DB::TCPHandler::runImpl() @ 0x0000000013aa32c9 in /usr/bin/clickhouse
DB::TCPHandler::run() @ 0x0000000013ab7c79 in /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start() @ 0x000000001658ffb4 in /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run() @ 0x00000000165911b1 in /usr/bin/clickhouse
Poco::PooledThread::run() @ 0x000000001669c8c7 in /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001669ab5c in /usr/bin/clickhouse
? @ 0x00007f2f839bf609 in ?
? @ 0x00007f2f838e4133 in ?

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\django\db\backends\utils.py", line 87, in _execute
return self.cursor.execute(sql)
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_backend\driver\connection.py", line 105, in execute
super().execute(operation, parameters)
File "C:\Users\vignesh\AppData\Local\Programs\Python\Python38\lib\site-packages\clickhouse_driver\dbapi\cursor.py", line 117, in execute
raise OperationalError(orig)
clickhouse_driver.dbapi.errors.OperationalError: Code: 57.
DB::Exception: Table django_local.django_migrations already exists. Stack trace:

DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000000cdd11b7 in /usr/bin/clickhouse
DB::Exception::Exception<String&, String, String>(int, FormatStringHelperImpl<std::type_identity<String&>::type, std::type_identity::type, std::type_identity::type>, String&, String&&, String&&) @ 0x0000000006c9791e in /usr/bin/clickhouse
DB::InterpreterCreateQuery::doCreateTable(DB::ASTCreateQuery&, DB::InterpreterCreateQuery::TableProperties const&, std::unique_ptr<DB::DDLGuard, std::default_deleteDB::DDLGuard>&) @ 0x00000000125e309f in /usr/bin/clickhouse
DB::InterpreterCreateQuery::createTable(DB::ASTCreateQuery&) @ 0x00000000125d9ddf in /usr/bin/clickhouse
DB::InterpreterCreateQuery::execute() @ 0x00000000125e725e in /usr/bin/clickhouse
DB::executeQueryImpl(char const*, char const*, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x0000000012be45f5 in /usr/bin/clickhouse
DB::executeQuery(String const&, std::shared_ptrDB::Context, bool, DB::QueryProcessingStage::Enum) @ 0x0000000012bde2fe in /usr/bin/clickhouse
DB::TCPHandler::runImpl() @ 0x0000000013aa32c9 in /usr/bin/clickhouse
DB::TCPHandler::run() @ 0x0000000013ab7c79 in /usr/bin/clickhouse
Poco::Net::TCPServerConnection::start() @ 0x000000001658ffb4 in /usr/bin/clickhouse
Poco::Net::TCPServerDispatcher::run() @ 0x00000000165911b1 in /usr/bin/clickhouse
Poco::PooledThread::run() @ 0x000000001669c8c7 in /usr/bin/clickhouse
Poco::ThreadImpl::runnableEntry(void*) @ 0x000000001669ab5c in /usr/bin/clickhouse
? @ 0x00007f2f839bf609 in ?
? @ 0x00007f2f838e4133 in ?

Versions

ClickHouse server version. Version can be obtained by running SELECT version() query. - 23.10.1.1976
Python version. 3.8
Clickhouse-driver version. - clickhouse-driver==0.2.6
Django version. 4.2
Django clickhouse backend version. - django-clickhouse-backend==1.1.3

The text was updated successfully, but these errors were encountered:

jayvynl · 2023-11-11T08:53:06Z

Sorry, I can't reproduce this issue. My steps are:

Create project
```
django-admin startproject aaa
```
Add clickhouse database setting
Migrate successfully

Maybe I miss something? My dependencies are:

ClickHouse server version 23.6.2.18
Python version 3.8.18
Clickhouse-driver version 0.2.6
Django version 4.2.7
Django clickhouse backend version 1.1.3

vigneshshettyin · 2023-11-12T17:24:45Z

Are you using default database as Clickhouse? @jayvynl

vigneshshettyin · 2023-11-13T05:02:53Z

Also, does [django-clickhouse-backend] package support filter for array fields?

jayvynl · 2023-11-13T10:15:55Z

Are you using default database as Clickhouse? @jayvynl

Yes

jayvynl · 2023-11-13T10:18:11Z

Also, does [django-clickhouse-backend] package support filter for array fields?

Yes, see docs/Fields.md

vigneshshettyin · 2023-11-17T10:03:46Z

Could you please share the django project code where you tested, for me and my colleague this is still not working?

We are getting the same error

clickhouse_driver.dbapi.errors.OperationalError: Code: 57.
DB::Exception: Table default.django_migrations already exists. Stack trace:

@jayvynl

jayvynl · 2023-11-19T14:37:33Z

Could you please share the django project code where you tested, for me and my colleague this is still not working?

We are getting the same error
clickhouse_driver.dbapi.errors.OperationalError: Code: 57.
DB::Exception: Table default.django_migrations already exists. Stack trace:
@jayvynl

All codes are generated default by django 4.2.7 with django-admin startproject aaa .

Please try thease steps to check if you can get error:

python3.8 -m venv venv
source venv/bin/activate
pip install django-clickhouse-backend
django-admin startproject aaa
# edit `aaa/aaa/settings.py` and add your clickhouse database settings.
cd aaa
python manage.py migrate

ernestasga · 2023-11-23T10:56:03Z

Can confirm this issue. I'm unable to setup the database, even after dropping the "django_migrations" table, I always receive this error.

clickhouse_driver.errors.ServerException: Code: 57.
DB::Exception: Table default.django_migrations already exists. Stack trace:
....
django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (Code: 57.
DB::Exception: Table default.django_migrations already exists. Stack trace:
....

This is the settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.environ['POSTGRES_NAME'],
        'USER': os.environ['POSTGRES_USER'],
        'PASSWORD': os.environ['POSTGRES_PASSWORD'],
        'HOST': os.environ.get('POSTGRES_HOST', 'postgres'),
        'PORT': os.environ.get('POSTGRES_PORT', '5432'),
    },
    'clickhouse': {
        'ENGINE': 'clickhouse_backend.backend',
        'NAME': os.environ['CLICKHOUSE_NAME'],
        'USER': os.environ['CLICKHOUSE_USER'],
        'PASSWORD': os.environ['CLICKHOUSE_PASSWORD'],
        'HOST': os.environ.get('CLICKHOUSE_HOST', 'clickhouse'),
        'PORT': os.environ.get('CLICKHOUSE_PORT', '9440'),
        'OPTIONS': {
            'secure': True
        }
    }
}
DATABASE_ROUTERS = ["app.dbrouters.ClickHouseRouter"]

@vigneshshettyin Have you found a workaround by any chance?

vigneshshettyin · 2023-11-23T11:28:41Z

Nope. The option available is unmanaged tables. @ernestasga

ernestasga · 2023-11-23T13:33:00Z

I was able to fix it by modifying table creation query. Feel free to use my fork until my PR is accepted and a new release is pushed.

vigneshshettyin · 2023-11-23T14:48:52Z

Thanks for suggesting the fix, I will create a monkey patch for the same until your PR is merged.
@ernestasga

vigneshshettyin · 2023-11-24T05:09:23Z

Observed the same issue with constraints, It's not checking if the same constraint exists.

Steps to achieve

Create a model Event as below

from django.db.models import CheckConstraint, IntegerChoices, Q
from django.utils import timezone
from clickhouse_backend import models
class Event(models.ClickhouseModel):
    class Action(IntegerChoices):
        PASS = 1
        DROP = 2
        ALERT = 3
    ip = models.GenericIPAddressField(default="::")
    ipv4 = models.IPv4Field(default="127.0.0.1")
    ip_nullable = models.GenericIPAddressField(null=True)
    port = models.UInt16Field(default=0)
    protocol = models.StringField(default="", low_cardinality=True)
    content = models.JSONField(default=dict)
    timestamp = models.DateTime64Field(default=timezone.now)
    created_at = models.DateTime64Field(auto_now_add=True)
    action = models.EnumField(choices=Action.choices, default=Action.PASS)

    class Meta:
        ordering = ["-timestamp"]
        engine = models.MergeTree(
            primary_key="timestamp",
            order_by=("timestamp", "id"),
            partition_by=models.toYYYYMMDD("timestamp"),
            index_granularity=1024,
            index_granularity_bytes=1 << 20,
            enable_mixed_granularity_parts=1,
        )
        indexes = [
            models.Index(
                fields=["ip"],
                name="ip_set_idx",
                type=models.Set(1000),
                granularity=4
            ),
            models.Index(
                fields=["ipv4"],
                name="ipv4_bloom_idx",
                type=models.BloomFilter(0.001),
                granularity=1
            )
        ]
        constraints = (
            CheckConstraint(
                name="port_range",
                check=Q(port__gte=0, port__lte=65535),
            ),
        )

1.python manage.py makemigrations - PASS
2.python manage.py migrate - PASS
3.python manage.py migrate - FAIL

If migrations is already applied and we try to migrate again, due to pre-existing same constraints it will FAIL.

ernestasga · 2023-11-24T09:50:01Z

@jayvynl have you tried cloud hosted instane? I'm getting this error when connecting to cloud on port 9440, but if I spin up a self hosted server with docker, running on port 9000 the error disappears.

ernestasga · 2023-11-24T11:36:46Z

I believe it may be server version incompatibility, since cloud is running v23.9

jayvynl · 2023-11-24T14:52:09Z

@ernestasga @vigneshshettyin I have write a test project for this issue, please test and verify.

ernestasga · 2023-11-24T15:31:28Z

@jayvynl It is indeed version incompatibility. Cloud is running version 23.9. If you pull image clickhouse/clickhouse-server:23.9 the issue can be reproduced.

jayvynl · 2023-11-25T03:29:29Z

@jayvynl It is indeed version incompatibility. Cloud is running version 23.9. If you pull image clickhouse/clickhouse-server:23.9 the issue can be reproduced.

Well done @ernestasga , the version does matter, I should have paid attention to the original version mentioned by @vigneshshettyin , however I didn't.

I have checked ClickHouse 23.9 changelog. The type of field information_schema.tables.table_type changed from Enum8 to String. This change makes inspecting database not working properly.

Thank you @ernestasga @vigneshshettyin. I will fix it soon.

vigneshshettyin · 2023-11-25T05:58:08Z

Waiting for the fix to be pushed @jayvynl !

vigneshshettyin added the bug Something isn't working label Nov 4, 2023

vigneshshettyin assigned jayvynl Nov 4, 2023

jayvynl added a commit that referenced this issue Nov 25, 2023

fix(introspection): fix #57

b26d39d

jayvynl closed this as completed in 0946ff4 Nov 25, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

python manage.py migrate not working #57

python manage.py migrate not working #57

vigneshshettyin commented Nov 4, 2023

jayvynl commented Nov 11, 2023 •

edited

vigneshshettyin commented Nov 12, 2023 •

edited

vigneshshettyin commented Nov 13, 2023

jayvynl commented Nov 13, 2023

jayvynl commented Nov 13, 2023

vigneshshettyin commented Nov 17, 2023

jayvynl commented Nov 19, 2023

ernestasga commented Nov 23, 2023 •

edited

vigneshshettyin commented Nov 23, 2023

ernestasga commented Nov 23, 2023

vigneshshettyin commented Nov 23, 2023

vigneshshettyin commented Nov 24, 2023

ernestasga commented Nov 24, 2023

ernestasga commented Nov 24, 2023

jayvynl commented Nov 24, 2023 •

edited

ernestasga commented Nov 24, 2023

jayvynl commented Nov 25, 2023 •

edited

vigneshshettyin commented Nov 25, 2023

python manage.py migrate not working #57

python manage.py migrate not working #57

Comments

vigneshshettyin commented Nov 4, 2023

jayvynl commented Nov 11, 2023 • edited

vigneshshettyin commented Nov 12, 2023 • edited

vigneshshettyin commented Nov 13, 2023

jayvynl commented Nov 13, 2023

jayvynl commented Nov 13, 2023

vigneshshettyin commented Nov 17, 2023

jayvynl commented Nov 19, 2023

ernestasga commented Nov 23, 2023 • edited

vigneshshettyin commented Nov 23, 2023

ernestasga commented Nov 23, 2023

vigneshshettyin commented Nov 23, 2023

vigneshshettyin commented Nov 24, 2023

ernestasga commented Nov 24, 2023

ernestasga commented Nov 24, 2023

jayvynl commented Nov 24, 2023 • edited

ernestasga commented Nov 24, 2023

jayvynl commented Nov 25, 2023 • edited

vigneshshettyin commented Nov 25, 2023

jayvynl commented Nov 11, 2023 •

edited

vigneshshettyin commented Nov 12, 2023 •

edited

ernestasga commented Nov 23, 2023 •

edited

jayvynl commented Nov 24, 2023 •

edited

jayvynl commented Nov 25, 2023 •

edited