Iterate on Java codegen #299

jmchilton · 2019-11-26T19:52:55Z

Build maven project containing Java objects for record types, capable of validation, and that can generate JavaDocs with schema docs contained within.

Missing stuff tracked #303.

codecov · 2019-11-26T21:28:29Z

Codecov Report

Merging #299 into master will increase coverage by 2.4%.
The diff coverage is 63.63%.

@@            Coverage Diff            @@
##           master     #299     +/-   ##
=========================================
+ Coverage   59.64%   62.05%   +2.4%     
=========================================
  Files          14       14             
  Lines        2622     2630      +8     
  Branches      727      728      +1     
=========================================
+ Hits         1564     1632     +68     
+ Misses        890      827     -63     
- Partials      168      171      +3

Impacted Files	Coverage Δ
schema_salad/codegen_base.py	`94.59% <100%> (+14.59%)`	⬆️
schema_salad/main.py	`56.49% <100%> (+0.49%)`	⬆️
schema_salad/codegen.py	`91.02% <33.33%> (+74.8%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4c83be8...6515752. Read the comment docs.

tetron · 2019-11-26T21:41:00Z

One thing you will want to figure out early on is how best to represent sum types in Java, eg a run field can be an embedded Workflow or a CommandLineTool or a ExpressionTool or a URI string. Returning Object and casting everywhere is going to get tedious very quickly.

jmchilton · 2019-11-27T02:52:43Z

Seems like Object would at least be a place to start 😅. The yak I'm shaving with this project currently I think would mostly just benefit from the UnionLoader loading types and verifying things. Validation still happens even if the interface to using the beans is crap.

Still... if I were going to put time into this... hmm... my first thought is it may be worth building UnionOf2<T1, T2>, UnionOf3<T1, T2, T3> .... and UnionOf2Loader, UnionOf3Loader, ....

class UnionOf2<T1, T2> {
   private Object value;
   public Object get();
   private int index; // track type of object by index 
   public boolean is1();
   public boolean is2();
   public T1 get1();
   public T2 get2();
   public Optional<T1> getOptional1();
   public Optional<T2> getOptional2();
}

The type definitions on the beans could become very ugly but may be somewhat informative. Certainly the potential for a lot more type safety also.

Nervous about investing too much time here though.

lgtm-com · 2019-11-27T05:23:05Z

This pull request introduces 1 alert when merging d23d056 into 4a3f8ca - view on LGTM.com

new alerts:

1 for Unnecessary pass

lgtm-com · 2019-11-27T06:57:28Z

This pull request introduces 2 alerts when merging a04e02d into 4a3f8ca - view on LGTM.com

new alerts:

1 for Unnecessary pass
1 for Unused import

tetron · 2019-11-27T16:38:23Z

@jmchilton you're a beast, thank you so much for working on this

jmchilton · 2019-11-28T00:52:37Z

I got tripped up because schema-salad-tool and codegen modules treat fields with default but not explicitly annotated as optional differently. Probably CWL specs were better about this and so it hasn't been encountered. Eliminating those from my schema - this works! It really validates my relatively complex schema.

jmchilton · 2019-11-28T02:49:56Z

Now with encoded Java formatting (https://github.com/google/google-java-format) and JavaDocs.

lgtm-com · 2019-11-28T02:54:57Z

This pull request introduces 4 alerts and fixes 1 when merging daedf25 into 9d1c5e1 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unused import
1 for Unreachable code

fixed alerts:

1 for Unused local variable

lgtm-com · 2019-11-28T14:09:34Z

This pull request introduces 3 alerts and fixes 1 when merging ead6ea0 into 9d1c5e1 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unreachable code

fixed alerts:

1 for Unused local variable

lgtm-com · 2019-11-28T14:56:17Z

This pull request introduces 3 alerts when merging cfbe9e2 into 4806936 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unreachable code

jmchilton · 2019-11-28T15:03:17Z

mr-c · 2019-11-28T15:11:01Z

@jmchilton Super cool! Can you add a main method?

mr-c · 2019-11-28T15:13:27Z

And while I'm making wishes, it would be nice if the schema docs became javadocs too.

I tried your code with both cwl v1.0 and v1.1 's schemas, no crashes :-)

lgtm-com · 2019-11-28T15:16:26Z

This pull request introduces 3 alerts when merging 7d8c67b into 4806936 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unreachable code

jmchilton · 2019-11-28T15:16:43Z

A main method that validates a document I assume? Sounds easy enough.

Placing the schema docs in the javadocs is a cool idea! I'm not sure...

jmchilton · 2019-11-28T19:43:20Z

I registered the requested validator as the entry point for the jar and I added the maven assembly plugin to build a self contained jar variant that is good for a standalone demo:

$ mvn assembly:single
$ java -jar target/gxformat2-0.0.1-SNAPSHOT-jar-with-dependencies.jar ../test-workflow.yml
Exception in thread "main" org.galaxyproject.gxformat2.gxformat2.utils.ValidationException: Failed to match union type
  Trying 'RecordField'
    the `inputs` field is not valid because:
      Failed to match union type
        Expected object with Java type of java.util.List but got java.util.HashMap
        Trying 'RecordField'
          the `type` field is not valid because:
            Failed to match union type
              Expected null
              Expected one of [File, data, collection]
  Expected object with Java type of java.util.List but got java.util.LinkedHashMap
	at org.galaxyproject.gxformat2.gxformat2.utils.UnionLoader.load(UnionLoader.java:31)
	at org.galaxyproject.gxformat2.gxformat2.utils.Loader.documentLoad(Loader.java:41)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:19)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:85)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:73)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:65)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:31)
	at org.galaxyproject.gxformat2.gxformat2.utils.RootLoader.loadDocument(RootLoader.java:48)
	at org.galaxyproject.gxformat2.gxformat2.utils.Validator.main(Validator.java:12)

lgtm-com · 2019-11-28T19:48:21Z

This pull request introduces 3 alerts when merging 5a6add1 into 4c83be8 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unreachable code

lgtm-com · 2019-11-28T20:04:06Z

This pull request introduces 3 alerts when merging 074722b into 4c83be8 - view on LGTM.com

new alerts:

2 for Unused named argument in formatting call
1 for Unreachable code

arguably should be constructors?

jmchilton · 2019-12-02T04:48:53Z

Wasn't my favorite design decision but I resolved a bunch of the type problems by just merging the JavaTypeDef changes back into TypeDef as optional attributes. Fixed most of the remaining type issues.

Added parts of the schema salad docs into the JavaDocs - class, interface, and getters:

lgtm-com · 2019-12-02T04:54:57Z

This pull request introduces 1 alert and fixes 2 when merging f50a87f into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T16:23:42Z

This pull request introduces 1 alert and fixes 2 when merging 5841a80 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T16:45:24Z

This pull request introduces 1 alert and fixes 2 when merging 57f9252 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T17:17:47Z

This pull request introduces 1 alert and fixes 2 when merging 284c9e7 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T17:38:30Z

This pull request introduces 1 alert and fixes 2 when merging 7c0ac87 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T20:21:57Z

This pull request introduces 1 alert and fixes 2 when merging e6ca905 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

lgtm-com · 2019-12-02T20:34:43Z

This pull request introduces 1 alert and fixes 2 when merging 6515752 into 4c83be8 - view on LGTM.com

new alerts:

1 for Unreachable code

fixed alerts:

2 for Unused named argument in formatting call

mr-c · 2019-12-03T16:19:29Z

schema_salad/main.py

+        "--codegen-target",
+        type=str,
+        default=None,
+        help="Defaults to sys.stdout for python (and ./ for hidden Java codegen)",


"Generate classes in target language, currently supported: python"

I'll add java to the list and remove the word hidden in a follow up.

mr-c · 2019-12-03T16:19:58Z

schema_salad/main.py

+        type=str,
+        metavar="directory",
+        default=None,
+        help="Directory of example documents for test case generation (Java only).",


Is there an example of how to use this?

https://github.com/common-workflow-language/schema_salad/blob/master/schema_salad/tests/test_java_codegen.py#L23

I wanted to allow valid and invalid document tests but I never got that far. Currently just assumes all the files in the examples directory are valid documents of this type.

mr-c · 2019-12-03T16:22:22Z

W00t! Shall I merge?

jmchilton · 2019-12-03T16:25:08Z

I have followups in the pipeline - but I'm not touching anything while this is such a pretty shade of green 😆. I'd go ahead an merge now.

See comment #299 (comment)

See comment common-workflow-language#299 (comment)

See comment #299 (comment)

jmchilton force-pushed the java_codegen_iter branch from 142aef9 to daedf25 Compare November 28, 2019 02:46

jmchilton force-pushed the java_codegen_iter branch from cfbe9e2 to 7d8c67b Compare November 28, 2019 15:01

jmchilton added 9 commits November 28, 2019 15:35

java codegen - setup a maven project (allows tests, deps, etc)

f87ab19

java codegen - utils for Loader interface and LoadingOptions

1b65070

java codegen - stup out all Loader impls

883d8a5

java codegen - instantiate loaders for all type declarations

7485952

java codegen - override fromDoc on Savable instances.

47777af

arguably should be constructors?

java codegen - implement declare_id_field

63932e7

java codegen - more var prefixing to defend against collision.

2da69cb

java codegen - implement root document loaders

49cf604

java codegen - test case to verify YAML loading

09f5cce

java codegen - more type fixes

f50a87f

jmchilton force-pushed the java_codegen_iter branch from 6f1311d to f50a87f Compare December 2, 2019 04:46

jmchilton added 2 commits December 2, 2019 10:57

java codegen - Emulate Python/salad unsplit on _:UUID URIs.

4134cbb

java codegen - randomly change things I don't understand to fix types

f296dba

jmchilton added 2 commits December 2, 2019 12:07

java codegen - fix more lines too long

278e268

java codegen - more tests (metascheme test)

284c9e7

jmchilton force-pushed the java_codegen_iter branch from 57f9252 to 284c9e7 Compare December 2, 2019 17:07

java codegen - Fix pkg_resource hack for Python 3.8 typing

e6ca905

jmchilton force-pushed the java_codegen_iter branch from 7c0ac87 to e6ca905 Compare December 2, 2019 20:12

java codegen - include plumbing in MANIFEST

6515752

jmchilton changed the title ~~[WIP] Iterate on Java codegen~~ Iterate on Java codegen Dec 2, 2019

mr-c reviewed Dec 3, 2019

View reviewed changes

mr-c merged commit 03a1f23 into common-workflow-language:master Dec 3, 2019

jmchilton added a commit that referenced this pull request Dec 3, 2019

java codegen - disable OneOrListOf types by default

2af5386

See comment #299 (comment)

jmchilton added a commit to jmchilton/schema_salad that referenced this pull request Dec 3, 2019

java codegen - disable OneOrListOf types by default

9e5badf

See comment common-workflow-language#299 (comment)

jmchilton added a commit that referenced this pull request Dec 3, 2019

java codegen - remove stray import not cleaned up in #299.

abd582f

jmchilton mentioned this pull request Dec 3, 2019

Random assorted cleanups for Java codegen #304

Merged

jmchilton added a commit that referenced this pull request Dec 3, 2019

java codegen - disable OneOrListOf types by default

d822b2b

See comment #299 (comment)

jmchilton added a commit that referenced this pull request Dec 3, 2019

java codegen - remove stray import not cleaned up in #299.

8650cc3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Iterate on Java codegen #299

Iterate on Java codegen #299

jmchilton commented Nov 26, 2019 •

edited

codecov bot commented Nov 26, 2019 •

edited

tetron commented Nov 26, 2019

jmchilton commented Nov 27, 2019

lgtm-com bot commented Nov 27, 2019

lgtm-com bot commented Nov 27, 2019

tetron commented Nov 27, 2019

jmchilton commented Nov 28, 2019

jmchilton commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Nov 28, 2019

mr-c commented Nov 28, 2019

mr-c commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Nov 28, 2019

jmchilton commented Nov 28, 2019 •

edited

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

mr-c Dec 3, 2019

jmchilton Dec 3, 2019

mr-c Dec 3, 2019

jmchilton Dec 3, 2019

mr-c commented Dec 3, 2019

jmchilton commented Dec 3, 2019

Iterate on Java codegen #299

Iterate on Java codegen #299

Conversation

jmchilton commented Nov 26, 2019 • edited

codecov bot commented Nov 26, 2019 • edited

Codecov Report

tetron commented Nov 26, 2019

jmchilton commented Nov 27, 2019

lgtm-com bot commented Nov 27, 2019

lgtm-com bot commented Nov 27, 2019

tetron commented Nov 27, 2019

jmchilton commented Nov 28, 2019

jmchilton commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Nov 28, 2019

mr-c commented Nov 28, 2019

mr-c commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Nov 28, 2019

jmchilton commented Nov 28, 2019 • edited

lgtm-com bot commented Nov 28, 2019

lgtm-com bot commented Nov 28, 2019

jmchilton commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

lgtm-com bot commented Dec 2, 2019

mr-c Dec 3, 2019

Choose a reason for hiding this comment

jmchilton Dec 3, 2019

Choose a reason for hiding this comment

mr-c Dec 3, 2019

Choose a reason for hiding this comment

jmchilton Dec 3, 2019

Choose a reason for hiding this comment

mr-c commented Dec 3, 2019

jmchilton commented Dec 3, 2019

jmchilton commented Nov 26, 2019 •

edited

codecov bot commented Nov 26, 2019 •

edited

jmchilton commented Nov 28, 2019 •

edited