Initial DDEX support (Entity Model)

[wwindcloud]

Hello there, as mentioned in pgfoundry this is my initial attempt to create a DDEX provider.

Since Npgsql can work without visual studio, it would be better to implement a thin layer supporting DDEX over Npgsql in visual studio when needed, and no dependencies attached when not need to. Thus, this merge is simply a separate VS package that can be installed or uninstalled using VS extension manager. You don't even need Npgsql to compile, but I add it to npgsql vs2012 sln for complete sake.

Right now, there's no data connection dialog support, only built-in VS dialog is used, so some properties might have problem showing itself upon changing the value, but it's just cosmetic, the underlying properties still works as expected.

Some part of Npgsql has been enhanced to support DDEX. Prime example is GetSchema API, it's useless if you cannot obtain primary keys and etc, so I add some more support.

StoreSchemaDefinitionVersion3 should be supported under NET45 and vs2012.

As said last time, there's bug somewhere preventing entity model from generating and I have pinpointed it now. It's a problem when the PrimitiveTypeKind is string or alike, your backend type conversion is smart and useful but it return a Byte[] as a result, but the visitedexpression engine only accept a String input to WriteSql, so things went haywire there. Right now, I fix it by convert utf8 encoded bytes to it's underlying ascii char, not sure if it's correct, but please check. I am a little curious here how this little error get unchecked.

There's still some known problem like DataSet cannot be added succussfully if you are using the data source wizard, i suspect it might be a bug in VS Designer. The Copy/Drag also not working as I am still figuring out why it said parameters are invalid.

Well, all in all, this should get a head start to let people and me to use EF using Npgsql with ease. Thanks.

[franciscojunior]

That's great, @wwindcloud !
Excellent work!

I'll test it and will let you know how it goes.

Regarding the byte[], it is a result of recent @glenebob Glen Parker's excellent work on performance improvements. We changed our internal data handling to be all byte[] to easy writing to networkstreams and this specific case went unnoticed. But as you noticed, a fix for this is really easy. The only change you would need to do is to use the UTF8 encoding, instead of ASCII. You could use the following method which is used when we need to get a string from byte[]:

BackendEncoding.UTF8Encoding.GetString(byte[]);

Just another questions, does this vs package work only on vs.net 2012 or does it work on 2010 or even on the 2008?

[wwindcloud]

No wonder, should be easy to fix similar cases around.

As I am using only vs 2012 right now, haven't bother checking with vs2010 or vs2008, but since I am using DDEX platform version 2.0, I assume they work as 2.0 is already supported from vs2008.

In the meantime I am trying to fix the copy/paste/drag/drop issue, it seems that the connectionstringbuilder is not conformant enough, somehow connection string were different from place to place as such VS cannot get back the original connection when an IDataObject get passed around containers, so that's why there's always an error pop out when pasting/dragging data tables/views from server explorer. Do you mind if I make changes to NpgsqlConnectionStringBuilder to add PropertyDescriptorAttributes, TypeConverterAttribute, etc to align with how Microsoft did theirs with SqlConnectionStringBuilder such that people can get/set Npgsql connection property with .net type conversion support on-the-fly especially on forms. Right now NpgsqlConnectionStringBuilder simply convert things internally and I think there might be some inconsistencies when .net/3rd component want to get/set some of the connection properties from Npgsql.

[franciscojunior]

If you find any other place where the byte[] approach is giving you problems, please let us know. :)

It's great to know you are using DDEX version which works since vs.net 2008. This will help us get more developers on board :)
I'll try getting it working on vs.net 2010. It is because I have the professional version only for 2010. And I remember that the express editions don't work with custom extensions.

Please, go ahead and make the changes. Your fixes will improve Npgsql compatibility and conformance. Those changes are always welcome! :)

[wwindcloud]

Ok, finally, most basic feature should be implemented by now. Please let me know if you find something wrong or bugs, I would like to see these integrated into the official version of Npgsql. Entity/Dataset/Query in VS should be working as expected.

The most drastic change is NpgsqlConnectionStringBuilder. Actually, the one implemented in Npgsql right now did not taken care the fact DbConnectionStringBuilder implement itself an ICustomTypeDescriptor, all public property type were getting exposed. 3rd party or VS can rely on ICustomTypeDescriptor and Reflection to get/set value from the builder. However, because DbConnectionStringBuilder will utimately convert every value into a string inside its' base container, getting the value back from base[keyword] will return you back a string instead of the original type. Therefore, those using reflection on SslMode, ProtocolVersion and System.Version will eventually lead to an exception because it doesn't know how to convert string back forth to from their original type. Base on the fact builders with the same connection string should have the same state, I have removed all custom property like _username, _database, _* etc and simply rely on the underneath DbConnectionStringBuilder collections.
By using attributes, NpgsqlConnectionStringBuilder can now dynamically offer display name/description in an dialog with localization support, different keyword can map to multiple other key strings and vice versa. Default value other than custom types can be defined in attributes. Add/Removing properties is far easier than before by simply adding/removing a property and attached the necessary attributes, all customization can be done inside the property. You can check how I did SslMode for custom typed properties. Only need to make two properties, one for real typed, one for string typed and the Class will taken care rest of the detail.

[franciscojunior]

Excellent, @wwindcloud !
I'm reviewing your changes and have some comments:

I noticed that on this line:
https://github.com/franciscojunior/Npgsql2/pull/67/files#diff-6daf78c3f8b7fabad257e702ed1ef44eR599
You reference the host property when handling a username missing problem.

Another thing I noticed is that NpgsqlConnectionStringBuilder.resx is now referencing system.forms version 4.0. This can give us problem compiling it to .net 3.5 and 2.0 for backward compatibility. Is it possible to keep referencing previous assembly versions? If not, I think we would need to check how to get different resx files compiled for different target framework versions. Maybe visual studio project files wizard @roji may help us! :)

I'm still checking your changes to NpgsqlConnectionStringBuilder and they seem to be very nice!

This PR conflicts with current master. Would you mind to rebase your code on master and resolve any conflicts you may find? This would help us a lot when merging your pull request. Thanks in advance.

[wwindcloud]

Yup, nice catch, that was a typo suppose to be username, not host. And revert to use 2.0 assembly as request and rebase my code to merge with your latest commit, hope things would be easier for you. Thanks.

[franciscojunior]

Thanks for the changes, now the merge applies cleanly.

But I noticed that NpgsqlConnectionStringBuilder doesn't compile on .net2.0 anymore. In fact, thanks to this line:

get { return props[keyword].GetValue(this); }

It only compiles on .net4.5 because of the GetValue(this) call.

Also, while testing on Xamarin Studio, I'm getting compilation errors in the lines:

foreach (Attribute attr in prop.GetCustomAttributes(typeof(NpgsqlConnectionStringKeywordAttribute)))

Because there is no method GetCustomAttributes which takes a single Type parameter. I had to add a second parameter with value false to make it compile past this point.

I still need to install vs.net 2012 to do more tests, and I don't know yet if it will allow me to create the extension. I remember express editions have some limitations about that.

But we would need to keep compatibility with .net 2.0. At least compilability with .net 3.5 (as we already don't have compilability with .net 2.0 although Npgsql can be run on .net 2.0)
Would you mind to try to make changes to NpgsqlConnectionStringBuilder to make it compatible with .Net 2.0? Or compilable with .net 3.5 and runnable with .net 2.0?

I think your modification will benefit both DDEX development as non-DDEX development as well. And for non-DDEX we would need compatibility with .net 2.0. But if the changes only concern about DDEX, I think we can work on some conditional compilation or something else. We would need to think about it.

Thank you very much in advance.

P.S.: I'll be out for some days and will be back on 24th. So I may not be able to comment until then.

[wwindcloud]

Yup, I haven't thought about .NET 2.0, made some changes, Npgsql should now able to be compiled under .NET 2.0. Also, I think DDEX can be done the same, only the resx need to be modified I think, can someone take a look as I don't know how to save a resx in vs2012 to an older version. Manually text editing perhaps???
Anyway, thanks for your feedback.

[franciscojunior]

Hi @wwindcloud !

When I merge your code, Npgsql testsuite is throwing errors. It says:

NpgsqlTests.CommandTests (TestFixtureSetUp):
SetUp : System.ArgumentException : key=value argument incorrect in ConnectionString
Parameter name: server

I think there may be something wrong with the keyword_mappings initialization.

I'm still checking what is going wrong, but if you have any idea about what the problem can be, it would be very helpful.

Thanks in advance.

[wwindcloud]

Ah! I did not run NpgsqlTest to test. Run it now and fix some of the bugs. I didn't realize the old version was case insensitive on Keywords, so just make all keyword uppercase then. I noticed there are at least 50 test case failed on my postgresql 9.2, I guess they need some work but at least NpgsqlConnectionStringBuilder is working and pass those tests now. Thanks.

[franciscojunior]

Excellent! Thanks for having a look at this.

It is strange that you are getting errors in the testsuite... You weren't
supposed to.

When I get home I'll check it out and I'll check those test errors too.

Thank you again for your help.
Em 27/10/2013 01:49, "wwindcloud" notifications@github.com escreveu:

Ah! I did not run NpgsqlTest to test. Run it now and fix some of the bugs.
I didn't realize the old version was case insensitive on Keywords, so just
make all keyword uppercase then. I noticed there are at least 50 test case
failed on my postgresql 9.2, I guess they need some work but at least
NpgsqlConnectionStringBuilder is working and pass those tests now. Thanks.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/67#issuecomment-27161556
.

[glenebob]

Hi guys,

I have some observations on NpgsqlConnectionStringBuilder changes...

1. Protocol is now internal, should be public.
2. SslMode is now internal, should be public.
3. PreloadReader defaults true, should be false.
4. UseExtendedTypes defaults true, used to be false.
5. Enlist defaults true, used to be false (although it doesn't appear to be used anywhere?)
6. Several properties getters throw "Keyword not supported" when no value is set, should return a default.

I think SslMode and Ssl should be connected. I assume Ssl exists for compatibly? Suggestion: Setting Ssl = true should set SslMode = Prefer, setting Ssl = false should set SslMode = Disable, setting SslMode = Prefer/Require should set Ssl = true and setting SslMode = Disable should set Ssl = false. Then we can simply remove the internal use of Ssl, and use SslMode only.

Protocal should default to Unknown, as it did in the past. PR #83 allows automatic version selection and fallback to work again.

-Glen

[wwindcloud]

Hello Glen,

The reason why I set Protocol and SslMode to internal because they were Enum type, and the internal IDictionary storage in DbConnectionString use string for everything, These can be solved by using a converter. But for some reason, IVsDataConnectionProperties doesn't use a converter when getting a property from NpgsqlConnectionStringBuilder and setting it to a DataColumn of a DataTable, the odd thing is the type of DataColumn is set according to the property types of DbConnectionStringBuilder, thus Enum for Protocol and SslMode. So by setting Protocol and SslMode to internal, it will make IVsDataConnectionProperties ignore these two properties and use protocolasstring and sslmodeasstirng instead, therefore eliminating the conversion problem.

People can use ProtocolAsString and SslModeAsString as replacement, type checking is still enforced internally as you can see in the code I explictly convert the string to Enum first before setting the internal property. However, if we really need Protocol and SslMode as an Enum type being public, I can customize IVsDataConnectionProperties to bypass the conversion problem.

3,4 and 5 can be solved by changing the DefaultValue attribute.

I am not sure about 6. If you apply DefaultValue to every property, you should be able to get everything unless you were passing a wrong keyword string, but let me check again on that.

Thanks for your feedback.

[kenjiuno]

Hi.

I'm interested in Npgsql's DDEX support.

I'm curious to know. Does this work relate to Npgsql Design time support preview? It is introduced at http://fxjr.blogspot.jp/2011/05/npgsql-design-time-support-preview.html

I'm privately extending its DDEX provider to fit to recent VS2012/2013.

[franciscojunior]

Hi, Kenji.

This isn't directly related to this post. The code from this post was a previous contribution. This pull request is a new implementation.

[kenjiuno]

Hi.
Thanks @franciscojunior I see that this is another one!

I want to test this DDEX support. @wwindcloud Could you please teach me how to test this solution.

I could build NpgsqlDdexProvider on VS2013 Pro (trial). And then I could install NpgsqlDdexProvider.vsix successfully.

I get a message when I'm making new data connection:

Failed to find or load the registered .Net Framework Data Provider.

https://drive.google.com/file/d/0BzIsP2o582nbUXFlZ3oyN1QweHM/edit?usp=sharing

Could you advise me what I need to do?

[franciscojunior]

Hi, Kenji!

I also tried to test the project but it says the project can't be open. I'm using vs2013 trial too. Do I need to install anything else? Thanks for any info.

[kenjiuno]

It seems it needs external sdk to open Visual Studio extensions project, please try this first:

Microsoft Visual Studio 2013 SDK
http://www.microsoft.com/en-us/download/details.aspx?id=40758

[kenjiuno]

Finally I could reach connection dialog.

My build steps, and installation steps:

• Prepare VS 2013 Pro.
• Microsoft Visual Studio 2013 SDK http://www.microsoft.com/en-us/download/details.aspx?id=40758
• Open Npgsql2012.sln.
• Select build configuration [Debug-net45] or such. At first, [Debug-net20] is selected. ddex can't be built.
• Open file "source.extension.vsixmanifest" from NpgsqlDdexProvider.
• [Install Targets] tab at left pane.
• [New] button at right side
  • Product Identifier: Microsoft.VisualStudio.Pro
  • Version Range: [12.0,13.0)
• Build NpgsqlDdexProvider.
• Run "Npgsql\NpgsqlDdexProvider\bin\Debug-net45\NpgsqlDdexProvider.vsix" to install it.
• Restart VS2013 Pro.

Trying

• Run VS2013 Pro.
• Show [Server Explorer]
• [Connection to database]
• Data Source: PostgreSQL Database

But I get error, when I click OK, after filling connection info.

「データ接続を追加できません。値の型が列の型と合いません。列Compatibleに<2.0.12.91>を格納できませんでした。必要な型はVersionです。」

It will be translated like:

"Unable to add data connection. Value type doesn't match column type. Column Compatible cannot store <2.0.12.91>. Required type is Version"

Anyone knows workaround for this issue?

[franciscojunior]

It seems it needs external sdk to open Visual Studio extensions project, please try this first:

Microsoft Visual Studio 2013 SDK
http://www.microsoft.com/en-us/download/details.aspx?id=40758

Great! I could open the project now. I thought I had installed that before. :)

How are you debugging the extension?

After checking this post: http://stackoverflow.com/questions/9281662/how-to-debug-visual-studio-extensions

I could debug it by installing Npgsql.dll and Mono.Security.dll to GAC and setting visual studio to start another experimental instance of visual studio itself where the extension would be installed. This way I didn't need to install the extension in the primary visual studio configuration.

I'm getting a lot of null reference exceptions which I'm checking. But I think we are very close to get it working. :)

@wwindcloud , do you have any tips for us?

[kenjiuno]

Hi.

I could debug it by installing Npgsql.dll and Mono.Security.dll to GAC and setting visual studio to start another experimental instance of visual studio itself where the extension would be installed.

Thanks. I could connect using connection dialog, and I could see some tables/views listed.

I'm getting a lot of null reference exceptions which I'm checking

It is problem on connection dialog?

I got the same problem, when I'm working on debugger.

Workaround: tweak ConvertTo to allow null input, at NpgsqlEnumConverter<T> in NpgsqlConnectionStringBuilder.cs,

            public override object ConvertTo(ITypeDescriptorContext context, System.Globalization.CultureInfo culture, object value, Type destinationType)
            {
                return (value != null) ? value.ToString() : String.Empty;
            }

In another case, I have trouble on data viewer of table/view.

It seems that VS failed to load an icon image. ArgumentException "Path format is invalid..." is thrown wrapped by TargetInvocationException.

I need a lot of inspections.

But I think we are very close to get it working. :)

Yes I think so too. :)

[kenjiuno]

System.IO.Path.GetExtension ate:

DataExplorer://127.0.0.1 (rice)/Tables/"rice"."public"."t1"

I don't understand why such a path is requested... Perhaps quotation marks cause trouble?

---

How to add breakpoint:

• Turn off "Enable Just My Code" from Tool→Option→Debugging
• Add new breakpoint at "System.IO.Path.GetExtension(string)"
• Make sure to load debug symbol for System.dll in Debug→Window→Modules

If you hit there, display Debug→Window→Local.

[franciscojunior]

Hi, Kenji!

I could get past the Connection String dialog, but I could only get the Schemata folder. I couldn't get the subitems like you have: pg_toast, public etc :(

I think it may be related to database permissions. I set up a simple user. Which permissions your database user have?

[wwindcloud]

Hello, sorry to have left for some time as I am busy with other things, I
will focus to finish this off so that everybody can get pgsql DDEX support
in VS.

Hi Kenji, good to hear somebody is interested in my work. Well, my DDEX is
not in anyway related to previous Npgsql Design time support, actually my
goal is to not build any custom design time support at all and rely on the
builtin support from VS or Entity Framework 6. That way, we don't need any
custom design interface but fed in the neccesary object to allow VS to let
it do the GUI work.

Ok, as for your errors, it's the problem of the VS builtin dialog box
mechanism, it doesn't know how to pass custom Enum<> values or Version
object, so that's why you get type not match error, and that's the reason
why I revamp the whole NpgsqlConnectionStringBuilder to use native types.
Well, I thought I fixed them, but seems like it got pop up in your case. I
will have to check.

On Mon, Dec 2, 2013 at 7:53 PM, kenjiuno notifications@github.com wrote:

Hi.

I'm interested in Npgsql's DDEX support.

I'm curious to know. Does this work relate to Npgsql Design time support
preview? It is introduced at
http://fxjr.blogspot.jp/2011/05/npgsql-design-time-support-preview.html

I'm privately extending its DDEX provider to fit to recent VS2012/2013.

—
Reply to this email directly or view it on GitHubhttps://github.com//pull/67#issuecomment-29612060
.

[franciscojunior]

Hello, sorry to have left for some time as I am busy with other things, I
will focus to finish this off so that everybody can get pgsql DDEX support
in VS.

Hi, @wwindcloud ! Welcome back.

Hi Kenji, good to hear somebody is interested in my work. Well, my DDEX is
not in anyway related to previous Npgsql Design time support, actually my
goal is to not build any custom design time support at all and rely on the
builtin support from VS or Entity Framework 6. That way, we don't need any
custom design interface but fed in the neccesary object to allow VS to let
it do the GUI work.

Ok, as for your errors, it's the problem of the VS builtin dialog box
mechanism, it doesn't know how to pass custom Enum<> values or Version
object, so that's why you get type not match error, and that's the reason
why I revamp the whole NpgsqlConnectionStringBuilder to use native types.
Well, I thought I fixed them, but seems like it got pop up in your case. I
will have to check.

That's excellent!

Please let us know if you have any questions about those changes.

[kenjiuno]

Hi!

Ok, as for your errors, it's the problem of the VS builtin dialog box
mechanism,

Ah, sorry. It is by my fault. I used different version of Npgsql. I have no problem now. :)

[samy2]

Hi @kenjiuno,

Ah, sorry. It is by my fault. I used different version of Npgsql. I have no problem now. :)

Can you be a bit more explicit ? I have exacly the same problem you described.
English translation of the error I have : "Unable to add data connection. Value type doesn't match column type. Column Compatible cannot store <2.1.1.0>. Required type is Version".

[kenjiuno]

Can you be a bit more explicit ? I have exacly the same problem you described.

I'm afraid I don't remember so much...

Perhaps we have brought another Npgsql.dll into GAC, from another DDEX support like Npgsql Design time support preview available for download!.

Try removing current Npgsql.dll from GAC.

[franciscojunior]

I remember when I was testing this patch that I left this field empty. Did you try that?

[kenjiuno]

At Microsoft Connect, Table data viewer hangs up on Npgsql DDEX provider report got reply.

Fix for this is available in Visual Studio 2013 Update 2 RC. Please install from http://www.microsoft.com/en-us/download/confirmation.aspx?id=42307
Thanks again for reporting this issue.

I'll test VS2013 Update 2 RC!

[kenjiuno]

Hi.

Can you be a bit more explicit ? I have exacly the same problem you described.
English translation of the error I have : "Unable to add data connection. Value type doesn't match column type. Column Compatible cannot store <2.1.1.0>. Required type is Version"

I have encountered your popup message today, after installing VS2013 Update 2 RC.

Here are tips to check.

• Check your connection dialog:
  
• Make sure to edit both x86 and x64's machine.config.
  VS2013 runs 64bit mode on x64 machine.
  C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config\machine.config
  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\Config\machine.config

  <system.data>
    <DbProviderFactories>
      <add name="Npgsql Data Provider" 
           invariant="Npgsql" 
           description=".Net Data Provider for PostgreSQL" 
           type="Npgsql.NpgsqlFactory, Npgsql, Version=2.0.12.91, Culture=neutral, PublicKeyToken=5d8b90d52f46fda7" 
           support="FF" />
    </DbProviderFactories>
  </system.data>

[kenjiuno]

Hi.

Can you be a bit more explicit ? I have exacly the same problem you described.
English translation of the error I have : "Unable to add data connection. Value type doesn't match column type. Column Compatible cannot store <2.1.1.0>. Required type is Version".

I knew why this happens...

• DbConnectionStringBuilder.TryGetValue returns all properties as System.String values. Compatible is returned as System.String.
• AdoDotNetRootObjectSelector.SelectObjects builds DataTable with properties queried from NpgsqlConnectionStringBuilder. Compatible is System.Version.
• AdoDotNetRootObjectSelector.SelectObjects calls TryGetValue. It failed to add a new DataRow, because
  System.String cannot be converted to System.Version.

This ArgumentException is thrown by DataTable/DataRow.

You can avoid this by implementing this inside NpgsqlConnectionStringBuilder.

        public override bool TryGetValue(string keyword, out object value) {
            try {
                value = GetValue(GetKey(keyword));
                return true;
            }
            catch (ArgumentException) {
                value = null;
                return false;
            }
        }

[kenjiuno]

Here is my branch.
kenjiuno/Npgsql@8afa577

It is branched from official v2.1.3.
And then I have merged my fix manually.
My fix is based on wwindcloud's PR.

I couldn't rebase correctly. 2.1 was so much changed from 2.0.

[franciscojunior]

That's excellent, @kenjiuno !!

Do you think it would be easy to create a pull request for master branch? This would be very helpful!

[samy2]

Here are tips to check.

Thank you very much @kenjiuno for these detailed informations.
You are right, I was using the official 2.1.1.0 npgsql version, and not the patched 2.0.12.91 version from wwindcloud. I wrongly assumed the pull request was commited in the 2.1.x branch.
Thank you for creating your own branch from 2.1.3 (I haven't tested it yet).

[kenjiuno]

Hi.

Thanks for helpful comment.

Yes, 2 branches are for recent 2.1 npgsql versions.

• kenjiuno/Npgsql@8afa577 is wrongly branched. It points released branch, not master branch. And it is not rebased work. So it cannot be merged into official branch. Please don't use it for developing.
• Fix #177. Add proper quoting to xml datatype. #212 Initial DDEX support : new pr #213 is a new branch which points to master branch. It can be merged even if some fixes are applied. Please use it.

[roji]

Closing as I guess this is no longer relevant. Please reopen if otherwise.