Schema: Parse column type #28

amorenoz · 2021-01-27T07:17:29Z

This PR parses the 'type' field in the column schema so that the type of each column can be easily derived.

It includes an example client that parses any schema file and prints a summary of the types of each column.

Currently, this PR depends on #26
Fixes: #27

amorenoz · 2021-01-27T07:21:31Z

As an example, using the print_schema example program on https://github.com/ovn-org/ovn/blob/master/ovn-sb.ovsschema outputs:

OVN_Southbound, (20.14.0)         
         Meter_Band                                                                                                                                                                   
                 burst_size => integer                                                                                                                                                          
                 action => enum (type: string): {[drop]}                                                                                                                                        
                 rate => integer                                                                                                                                                                
         Connection                                                                                                                                                                             
                 target => string                                                                                                                                                               
                 max_backoff => []integer (min: 0, max: 1)                                                                                                                                      
                 inactivity_probe => []integer (min: 0, max: 1)                                                                                                                                 
                 other_config => [string]string                                                                                                                                                 
                 status => [string]string                                                                                                                                                       
                 read_only => boolean                                                                                                                                                           
                 role => string                                                                                                                                                                 
                 external_ids => [string]string                                                                                                                                                 
                 is_connected => boolean                                                                                                                                                        
         IP_Multicast                                                                                                                                                                           
                 ip6_src => string                                                                                                                                                              
                 idle_timeout => []integer (min: 0, max: 1)                                                                                                                                     
                 seq_no => integer                                                                                                                                                              
                 enabled => []boolean (min: 0, max: 1)                                                                                                                                          
                 eth_src => string                                                                                                                                                              
                 ip4_src => string                                                                                                                                                              
                 table_size => []integer (min: 0, max: 1)                                                                                                                                       
                 query_interval => []integer (min: 0, max: 1)                                                                                                                                   
                 query_max_resp => []integer (min: 0, max: 1)                                                                                                                                   
                 datapath => uuid [Datapath_Binding (weak)]                                                                                                                                     
                 querier => []boolean (min: 0, max: 1)                                                                                                                                          
         Logical_DP_Group                                                                                                                                                                       
                 datapaths => [] [Datapath_Binding (weak)] (min: 0, max: -1)                                                                                                                    
         Datapath_Binding                                                                                                                                                                       
                 tunnel_key => integer                                                                                                                                                          
                 load_balancers => [] [Load_Balancer (weak)] (min: 0, max: -1)                                                                                                                  
                 external_ids => [string]string                                                                                                                                                 
         DHCP_Options                                                                                                                                                                           
                 name => string                                                                                                                                                                 
                 code => integer                                                                                                                                                                
                 type => enum (type: string): {[bool uint8 uint16 uint32 ipv4 static_routes str host_id domains]}                                                                               
         Chassis                                                                                                                                                                                
                 external_ids => [string]string                                                                                                                                                 
                 other_config => [string]string                                                                                                                                                 
                 transport_zones => []string (min: 0, max: -1)                                                                                                                                  
                 name => string                                                                                                                                                                 
                 hostname => string                                                                                                                                                             
                 encaps => [] [Encap ()] (min: 1, max: -1)                                                                                                                                      
                 vtep_logical_switches => []string (min: 0, max: -1)                                                                                                                            
                 nb_cfg => integer                                                                                                                                                              
         Encap 
                 type => enum (type: string): {[geneve stt vxlan]}
                 options => [string]string
                 ip => string
                 chassis_name => string
         RBAC_Role
                 name => string
                 permissions => [string]uuid
         Meter
                 name => string
                 unit => enum (type: string): {[kbps pktps]}
                 bands => [] [Meter_Band (strong)] (min: 1, max: -1)
         DNS
                 records => [string]string
                 datapaths => [] [Datapath_Binding ()] (min: 1, max: -1)
                 external_ids => [string]string
         Gateway_Chassis
                 name => string
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 priority => integer
                 external_ids => [string]string
                 options => [string]string
         Controller_Event
                 event_type => enum (type: string): {[empty_lb_backends]}
                 event_info => [string]string
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 seq_num => integer
         IGMP_Group
                 address => string
                 datapath => [] [Datapath_Binding (weak)] (min: 0, max: 1)
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 ports => [] [Port_Binding (weak)] (min: 0, max: -1)
         Load_Balancer
                 name => string
                 vips => [string]string
                 protocol => enum (type: string): {[tcp udp sctp]}
                 datapaths => [] [Datapath_Binding ()] (min: 0, max: -1)
                 external_ids => [string]string
         Logical_Flow
                 priority => integer
                 match => string
                 actions => string
                 external_ids => [string]string
                 logical_datapath => [] [Datapath_Binding ()] (min: 0, max: 1)
                 logical_dp_group => [] [Logical_DP_Group ()] (min: 0, max: 1)
                 pipeline => enum (type: string): {[ingress egress]}
                 table_id => integer
         SSL
                 bootstrap_ca_cert => boolean
                 ssl_protocols => string
                 ssl_ciphers => string
                 external_ids => [string]string
                 private_key => string
                 certificate => string
                 ca_cert => string
         Service_Monitor
                 ip => string
                 src_ip => string
                 external_ids => [string]string
                 protocol => enum (type: string): {[tcp udp]}
                 port => integer
                 logical_port => string
                 src_mac => string
                 status => enum (type: string): {[online offline error]}
                 options => [string]string
         BFD
                 src_port => integer
                 disc => integer
                 dst_ip => string
                 min_tx => integer
                 detect_mult => integer
                 external_ids => [string]string
                 options => [string]string
                 logical_port => string
                 min_rx => integer
                 status => enum (type: string): {[down init up admin_down]}
         SB_Global
                 options => [string]string
                 ipsec => boolean
                 nb_cfg => integer
                 external_ids => [string]string
                 connections => [] [Connection ()] (min: 0, max: -1)
                 ssl => [] [SSL ()] (min: 0, max: 1)
         Port_Group
                 name => string
                 ports => []string (min: 0, max: -1)
         Port_Binding
                 logical_port => string
                 tunnel_key => integer
                 type => string
                 gateway_chassis => [] [Gateway_Chassis (strong)] (min: 0, max: -1)
                 parent_port => []string (min: 0, max: 1)
                 virtual_parent => []string (min: 0, max: 1)
                 encap => [] [Encap (weak)] (min: 0, max: 1)
                 nat_addresses => []string (min: 0, max: -1)
                 datapath => uuid [Datapath_Binding ()]
                 tag => []integer (min: 0, max: 1)
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 up => []boolean (min: 0, max: 1)
                 ha_chassis_group => [] [HA_Chassis_Group (strong)] (min: 0, max: 1)
                 options => [string]string
                 mac => []string (min: 0, max: -1)
                 external_ids => [string]string
         MAC_Binding
                 ip => string
                 mac => string
                 datapath => uuid [Datapath_Binding ()]
                 logical_port => string
         DHCPv6_Options
                 code => integer
                 type => enum (type: string): {[ipv6 str mac]}
                 name => string
         RBAC_Permission
                 authorization => []string (min: 0, max: -1)
                 insert_delete => boolean
                 update => []string (min: 0, max: -1)
                 table => string
         HA_Chassis
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 priority => integer
                 external_ids => [string]string
         Chassis_Private
                 nb_cfg_timestamp => integer
                 external_ids => [string]string
                 name => string
                 chassis => [] [Chassis (weak)] (min: 0, max: 1)
                 nb_cfg => integer
         Address_Set
                 addresses => []string (min: 0, max: -1)
                 name => string
         Multicast_Group
                 datapath => uuid [Datapath_Binding ()]
                 name => string
                 tunnel_key => integer
                 ports => [] [Port_Binding (weak)] (min: 1, max: -1)
         HA_Chassis_Group
                 name => string
                 ha_chassis => [] [HA_Chassis (strong)] (min: 0, max: -1)
                 ref_chassis => [] [Chassis (weak)] (min: 0, max: -1)
                 external_ids => [string]string

schema.go

The way the column types are specified in RFC7047 is not directly unmarshallable by the json encoder. This patch adds the necessary structs and functions to manually unmarshal the schema definition. The result is the possibility of determining the exact time of golang structure needed to store each column type. Several approaches were considering when implementing this: A) Multiple rounds of json.Unmarshal storing the partial in in-struct json.rawMessage B) Single call to json.Unmarshall and manually decoding the rest of the message (using reflex to guess the type of incoming data) C) Multiple rounds of json.Unmarshal storing partials in private temporary structs. After discarting A) for it's high memory consumption, B) and C) were compared: B) has lower memory allocation needs but more LOC C) has higher memory allocation needs (though after GC, it's the same or less than B), and fewer LOC. This patch implements C) as a good balance between maintainablity and memory efficiency. Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

the special column "_uuid" is not on the schema object. If it is needed, generate the ColumnSchema on the fly Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

hzhou8 · 2021-03-22T03:38:54Z

client.go

@@ -18,15 +18,15 @@ import (
 // OvsdbClient is an OVSDB client
 type OvsdbClient struct {
 	rpcClient     *rpc2.Client
-	Schema        map[string]DatabaseSchema
+	Schema        map[string]*DatabaseSchema


This changes the interface. At least go-ovn will be broken due to this change. Is there a way to avoid interface change?

You're right. I'll try to avoid breaking the API in #30

schema.go

hzhou8 · 2021-03-22T07:44:39Z

Thanks @amorenoz! Please take a look at my comments. In addition, apart from the example, could you add some test cases?

amorenoz · 2021-03-22T07:56:37Z

Sorry @hzhou8, This PR has been superseded by #30 and it's been a while since I don't update this one.
I'll should have closed it as @vtolstov suggested.
However, some of your comments are still valid for #30 so I'll update that PR accordingly.

hzhou8 · 2021-03-22T08:04:54Z

@amorenoz ok, I was reviewing in chronological order, so I thought #30 depends on this one. Let's close this one and I will review #30 this week.

amorenoz mentioned this pull request Feb 2, 2021

Proposed api interface changes eBay/go-ovn#29

Open

vtolstov mentioned this pull request Feb 3, 2021

send raw table update to clients #14

Closed

vtolstov suggested changes Feb 3, 2021

View reviewed changes

schema.go Outdated Show resolved Hide resolved

schema.go Outdated Show resolved Hide resolved

amorenoz force-pushed the rfe/schema-support branch 3 times, most recently from 4ac8ac7 to a6afc84 Compare February 11, 2021 18:54

amorenoz mentioned this pull request Feb 11, 2021

Add native bindings and API #30

Merged

amorenoz added 4 commits February 17, 2021 18:18

schema: add getColumn function

fa06e5d

the special column "_uuid" is not on the schema object. If it is needed, generate the ColumnSchema on the fly Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

example: move example to it's own directory

1899655

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

example: add a simple program to print a schema

f94feaa

Signed-off-by: Adrian Moreno <amorenoz@redhat.com>

amorenoz force-pushed the rfe/schema-support branch from a6afc84 to f94feaa Compare February 17, 2021 17:32

vtolstov approved these changes Mar 11, 2021

View reviewed changes

hzhou8 suggested changes Mar 22, 2021

View reviewed changes

hzhou8 closed this Mar 22, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schema: Parse column type #28

Schema: Parse column type #28

amorenoz commented Jan 27, 2021

amorenoz commented Jan 27, 2021

hzhou8 Mar 22, 2021 •

edited

amorenoz Mar 22, 2021

hzhou8 commented Mar 22, 2021

amorenoz commented Mar 22, 2021

hzhou8 commented Mar 22, 2021

Schema: Parse column type #28

Schema: Parse column type #28

Conversation

amorenoz commented Jan 27, 2021

amorenoz commented Jan 27, 2021

hzhou8 Mar 22, 2021 • edited

Choose a reason for hiding this comment

amorenoz Mar 22, 2021

Choose a reason for hiding this comment

hzhou8 commented Mar 22, 2021

amorenoz commented Mar 22, 2021

hzhou8 commented Mar 22, 2021

hzhou8 Mar 22, 2021 •

edited