Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Question: can we filter metadata? #65

Closed
FixRM opened this issue Jun 1, 2020 · 10 comments
Closed

Question: can we filter metadata? #65

FixRM opened this issue Jun 1, 2020 · 10 comments

Comments

@FixRM
Copy link

FixRM commented Jun 1, 2020

Hello!

Thank you for a great tool! We use it to generate docs for Microsoft Common Data Service OData endpoint (formerly Dynamics CRM, formerly D365). This endpoint is generic and has quite big CSDL file (about 4 MB). Resulting JSON file is about 880 MB!!! Can you please suggest how we can cut some unused datasets and related stuff before generating OpenAPI file?

@xuzhg
Copy link
Contributor

xuzhg commented Jun 1, 2020

@FixRM Thanks for trying this tool.

  1. there are some settings that you can use to do filter. For example: if set EnableNavigationPropertyPath = false, there will be no navigation property path.

  2. IODataPathProvider has the "CanFilter" that is designed for your scenario, but so far, i don't have much requirement, so i haven't exposed it throw "ODataContext".

@FixRM
Copy link
Author

FixRM commented Jun 1, 2020

@xuzhg, thanks for quick reply. Can you please give me a sample of how to use IODataPathProvider?

@xuzhg
Copy link
Contributor

xuzhg commented Jun 1, 2020

@FixRM My design is that you can derive from "ODataPathProvider" and override this virtual method. However, "ODataPathProvider" is used in the ODataContext, that's an internal class.

I'd like to expose an interface in the setting class so developer can create an instance of "his own ODataPathProvider", but it's not supported yet.

@FixRM
Copy link
Author

FixRM commented Jun 1, 2020

I see. What other options do I have? I was thinking of filtering while loading XML nodes, but it seems tricky and my VS is dying when I try to load original file into it

@xuzhg
Copy link
Contributor

xuzhg commented Jun 1, 2020

  1. use the Capabilities annotation, but this way will change your CSDL
  2. I expose the Provider at the setting for you to set. Need some time to implement it.

@darrelmiller
Copy link
Member

If we removed examples as proposed here #68 it would help to reduce the document size.

Also, yaml produces a much more compact format than JSON.

@darrelmiller
Copy link
Member

darrelmiller commented Jul 4, 2020

@FixRM I think your best option would be to run the CSDL through an XSLT transform first to cut out the parts of the CSDL you don't want to process. We do some preprocessing of our CSDL this way using this approach https://github.com/microsoftgraph/msgraph-metadata/blob/master/transforms/csdl/preprocess_csdl.xsl

@FixRM
Copy link
Author

FixRM commented Jul 9, 2020

@darrelmiller, thank you for your suggestion. Can you please help me with the transform?

My CSDL looks like this:

<edmx:Edmx Version="4.0">
    <edmx:Reference Uri="http://vocabularies.odata.org/OData.Community.Keys.V1.xml">
    <edmx:Include Namespace="OData.Community.Keys.V1" Alias="Keys"/>
    <edmx:IncludeAnnotations TermNamespace="OData.Community.Keys.V1"/>
</edmx:Reference>
<edmx:Reference Uri="http://vocabularies.odata.org/OData.Community.Display.V1.xml">
    <edmx:Include Namespace="OData.Community.Display.V1" Alias="Display"/>
    <edmx:IncludeAnnotations TermNamespace="OData.Community.Display.V1"/>
</edmx:Reference>
<edmx:DataServices>
    <Schema Namespace="Microsoft.Dynamics.CRM" Alias="mscrm">
        <EntityType Name="crmbaseentity" Abstract="true"/>
        <EntityType Name="accountleads" BaseType="mscrm.crmbaseentity">
                    ***

I started with this one, but it generates same document.

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  
  <xsl:output method="xml" indent="yes"/>
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="Schema/EntityType[not(@Name='account')]" />
  
</xsl:stylesheet>

What am I doing wrong?

@FixRM
Copy link
Author

FixRM commented Jul 9, 2020

It was namespace problem. Correct transform:

<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
                xmlns:edmx="http://docs.oasis-open.org/odata/ns/edmx"
                xmlns:edm="http://docs.oasis-open.org/odata/ns/edm"
                xmlns="http://docs.oasis-open.org/odata/ns/edm">
  <xsl:strip-space elements="*"/><!-- Remove empty space after deletions. -->
  <xsl:output method="xml" indent="yes"/>
  <xsl:template match="@* | node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <!--Remove unused entities-->
  <!--<xsl:template match="edm:Schema/edm:EntityType[not(@Name='account')]" />-->

  <!--Remove unused entity sets-->
  <xsl:template match="edm:Schema/edm:EntityContainer/edm:EntitySet[not(@Name='accounts')]" />

  <!--Remove unused actions-->
  <!--<xsl:template match="edm:Schema/edm:Action" />-->

  <!--Remove unused types-->
  <!--<xsl:template match="edm:Schema/edm:ComplexType" />-->

  <!--Remove unused functions-->
  <!--<xsl:template match="edm:Schema/edm:Function" />-->

  <!--Remove unused enums-->
  <!--<xsl:template match="edm:Schema/edm:EnumType" />-->  

</xsl:stylesheet>

@FixRM FixRM closed this as completed Jul 9, 2020
@bkwdesign
Copy link

@FixRM My design is that you can derive from "ODataPathProvider" and override this virtual method. However, "ODataPathProvider" is used in the ODataContext, that's an internal class.

I'd like to expose an interface in the setting class so developer can create an instance of "his own ODataPathProvider", but it's not supported yet.

I also have an EDMX that produces over 6400 paths- and 1800+ schemas -my swagger result is only 17 MB, but I’m also interested in ways to pass in a schema-of-interest and have the rest ignored

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants