You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
<?phpnamespaceLLPhant\Embeddings\VectorStores\Doctrine;
useDoctrine\DBAL\Exception;
useDoctrine\DBAL\Platforms\AbstractPlatform;
useDoctrine\DBAL\Platforms\PostgreSQLPlatform;
useDoctrine\DBAL\Types\Type;
useDoctrine\Deprecations\Deprecation;
classVectorTypeextendsType
{
finalpublic const VECTOR = 'vector';
/** * @param mixed[] $column */publicfunctiongetSQLDeclaration(array$column, AbstractPlatform$platform): string
{
// getName is deprecated since doctrine/dbal 2.13 see: https://github.com/doctrine/dbal/issues/4749// BUT it is the most stable way to check if the platform is PostgreSQLPlatform in a lot of doctrine versions// so we will use it and add a check for the class name in case it is removed in the futureif (method_exists($platform, 'getName') && $platform->getName() !== 'postgresql') {
throwException::notSupported('VECTORs not supported by Platform.');
}
if (! isset($column['length'])) {
throwException::notSupported('VECTORs must have a length.');
}
if ($column['length'] < 1) {
throwException::notSupported('VECTORs must have a length greater than 0.');
}
if (! is_int($column['length'])) {
throwException::notSupported('VECTORs must have a length that is an integer.');
}
return sprintf('vector(%d)', $column['length']);
}
/** * @return float[] */publicfunctionconvertToPHPValue(mixed$value, AbstractPlatform$platform): array
{
if ($value === null) {
return [];
}
$value = is_resource($value) ? stream_get_contents($value) : $value;
if (! is_string($value)) {
throwException::notSupported('Error while converting VECTORs to PHP value.');
}
$convertedValue = explode(',', $value);
$floatArray = [];
foreach ($convertedValueas$singleConvertedValue) {
$floatArray[] = (float) $singleConvertedValue;
}
return$floatArray;
}
publicfunctionconvertToDatabaseValue(mixed$value, AbstractPlatform$platform): string
{
//If $value is not a float array throw an exceptionif (! is_array($value)) {
throwException::notSupported('VECTORs must be an array.');
}
returnVectorUtils::getVectorAsString($value);
}
publicfunctiongetName(): string
{
returnself::VECTOR;
}
}
finalclassVersion20231101073053extendsAbstractMigration
{
publicfunctionup(Schema$schema): void
{
$this->addSql('CREATE SEQUENCE snippet_id_seq INCREMENT BY 1 MINVALUE 1 START 1');
$this->addSql('CREATE TABLE snippet (id INT NOT NULL, embedding vector(1536) NOT NULL, PRIMARY KEY(id))');
}
publicfunctiondown(Schema$schema): void
{
$this->throwIrreversibleMigrationException();
}
}
From my understanding, because I added mapping_types: vector: array it should look like:
$this->addSql('CREATE SEQUENCE snippet_id_seq INCREMENT BY 1 MINVALUE 1 START 1');
$this->addSql('CREATE TABLE snippet (id INT NOT NULL, embedding vector(1536) NOT NULL, PRIMARY KEY(id))');
+ $this->addSql('COMMENT ON COLUMN snippet.embedding IS \'(DC2Type:array)\'');
If I manually add this line, all works good, but if I want to generate the diff again I get another migration over and over again:
finalclassVersion20231101073443extendsAbstractMigration
{
publicfunctionup(Schema$schema): void
{
$this->addSql('ALTER TABLE snippet ALTER embedding TYPE vector(1536)');
$this->addSql('COMMENT ON COLUMN snippet.embedding IS NULL');
}
publicfunctiondown(Schema$schema): void
{
$this->throwIrreversibleMigrationException();
}
}
The fact that your implementation of VectorType returns an array is is unknown to doctrine.
The comment is used to infer a Doctrine type when there are multiple doctrine types mapped on the same SQL type. The classical example is:
Consider an SQL type varchar(255), that type can be used as the Php's string, MyEnum (a backed enum that uses strings as cases) or any PHP type that can be serialized into a string with max 255 chrs.
When DBAL encounters a vector SQL type, needs to decide which DBAL Type assign to it. If a comment is found, it will try to find that type, if not it will find a default.
In this case, both GoetasVectorType and LLPhant\Embeddings\VectorStores\Doctrine\VectorType want to use a SQL vector, but DBAL needs to a way to decide which type to instantaite when an SQL column of type vector is found.
GoetasVectorType will have to override requiresSQLCommentHint. Columns that use GoetasVectorType will have a comment, columns that use LLPhant\Embeddings\VectorStores\Doctrine\VectorType will not.
When DBAL finds an SQL vector type will check for a comment, when found will try to instantiate the type in the comment. If no comment is found, will try to instantiate the type defined in the configs.
Using the latest bundle version and Postgres.
I have the following entity
VectorType
:the created migration:
From my understanding, because I added
mapping_types: vector: array
it should look like:$this->addSql('CREATE SEQUENCE snippet_id_seq INCREMENT BY 1 MINVALUE 1 START 1'); $this->addSql('CREATE TABLE snippet (id INT NOT NULL, embedding vector(1536) NOT NULL, PRIMARY KEY(id))'); + $this->addSql('COMMENT ON COLUMN snippet.embedding IS \'(DC2Type:array)\'');
If I manually add this line, all works good, but if I want to generate the diff again I get another migration over and over again:
cc @chr-hertel @goetas
The text was updated successfully, but these errors were encountered: