adding namespaces #325

soumilbaldota · 2022-08-21T18:04:07Z

BEGINRELEASENOTES

adding namespaces to julia code generation
ENDRELEASENOTES

tmadlener · 2022-08-23T11:35:41Z

Is this already fully functional? It seems that there are still a few bits missing, e.g. there are no new files generated yet.

…ruct in main namespace only

soumilbaldota · 2022-08-23T17:13:21Z

Is this already fully functional? It seems that there are still a few bits missing, e.g. there are no new files generated yet.

It is not fully functional as of yet. Yes adding the parent namespace is left. There was name conflicts occuring when using multiple modules at the same time, i.e. Main.x and Y.x would cause a name conflict and result in an error in the previous commit, so It took me some time to figure out how to avoid that.

tmadlener · 2022-08-24T08:05:21Z

It looks like things start to work again. Very nice. I just had a quick look at the unittests. Could you add one of the datatypes that use namespaces there as well? E.g.

podio/tests/datalayout.yaml

Lines 139 to 153 in a8efda9

    
           ex42::ExampleWithNamespace : 
        
             Description : "Type with namespace and namespaced member" 
        
             Author : "Joschka Lingemann" 
        
             Members: 
        
               - ex2::NamespaceStruct component // a component 
        
           ex42::ExampleWithARelation : 
        
             Description : "Type with namespace and namespaced relation" 
        
             Author : "Joschka Lingemann" 
        
             Members: 
        
              - float number // just a number 
        
             OneToOneRelations : 
        
              - ex42::ExampleWithNamespace ref // a ref in a namespace 
        
             OneToManyRelations : 
        
              - ex42::ExampleWithNamespace refs // multiple refs in a namespace

tmadlener

This looks quite good already I think. I have a few minor comments / questions inline.

tmadlener · 2022-08-24T08:08:40Z

python/templates/Constructor.jl.jinja2

+{% macro julia_parameters(params) %}
+{%- if params_jl -%}{
+{%- set comma = joiner(',') -%}
+{%- for par in params -%}
+{{ comma() }}Main.{{ par }}Struct
+{%- endfor -%}
+}{%- endif -%}
+{% endmacro %}


If I remember correctly a very similar macro already exists in the MutableStruct.jl.jinja2 template(?). Is it possible to combine both macros into one and place it into a macros/julia.jinja2 file and simply import that here? It would probably have to take more arguments to deal with the different pre- and suffixes.

tmadlener · 2022-08-24T08:11:31Z

python/templates/Collection.jl.jinja2

@@ -0,0 +1,2 @@
+include("{{ class.bare_type }}.jl")
+{{ class.bare_type }}Collection = Vector{ {{ class.bare_type }}{% if params_jl %}{ {% for relation in params_jl %} {{ relation }}, {% endfor %} }{% endif %} }


Suggested change

{{ class.bare_type }}Collection = Vector{ {{ class.bare_type }}{% if params_jl %}{ {% for relation in params_jl %} {{ relation }}, {% endfor %} }{% endif %} }

{{ class.bare_type }}Collection = Vector{ {{ class.bare_type }}{% if params_jl %}{ {{ params_jl | join(', ') }} }{% endif %} }

For consistency with how it is done in the constructors

Collection.jl.jinja2 was a mistake.

tmadlener · 2022-08-24T08:12:20Z

python/templates/JuliaCollection.jl.jinja2

+}{%- endif -%}
+{% endmacro %}
+
+{{ class.bare_type }}Collection = Vector{ Main.{{ class.bare_type }}Struct{{ julia_parameters(params_jl) }} }


What is the purpose of this, since there is also a Collection.jl.jinja2 template now?

Collection.jl.jinja2 was a mistake.

tmadlener · 2022-08-24T08:13:58Z

tests/unittest.jl

 include("datamodel/ExampleMC.jl")
+using .ExampleMCModule: ExampleMC


Is it possible to bring the ExampleMCCollection here via using .ExampleMCModule: ExampleMC, ExampleMCCollection?

simply doing include("ExampleMCCollection") works as of now which also makes the ExampleMC. We can call both ExampleMCCollection( ) and ExampleMC( ) directly.
Since we are bringing the Module out from its namespace anyway in the Collection, we do not explicitly need to include ExampleMC. so L1 and L2 can be removed.

python/templates/JuliaCollection.jl.jinja2

Co-authored-by: Thomas Madlener <thomas.madlener@desy.de>

python/podio_class_generator.py

Co-authored-by: Thomas Madlener <thomas.madlener@desy.de>

tmadlener · 2022-08-29T08:10:00Z

Hi @soumilbaldota, this looks like it solves things for the example datalayout of podio. I briefly tried to run it on the edm4hep definition and I have run into the following problem:

julia> include("edm4hep.jl")
WARNING: replacing module Vector3fModule.
WARNING: replacing module Vector3fModule.
[... more of those ...]
WARNING: ignoring conflicting import of Vector3dModule.Vector3d into Main
ERROR: LoadError: UndefVarError: ObjectID not defined
Stacktrace:
 [1] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/TrackerHitStruct.jl:5
 [2] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [3] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/TrackerHit.jl:1
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [5] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/edm4hep.jl:17
 [6] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [7] top-level scope
   @ REPL[1]:1
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/TrackerHitStruct.jl:5
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/TrackerHit.jl:1
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/edm4hep.jl:17

From a glance at the stacktrace it looks as if some of the necessary includes are missing for edm4hep. Could you have a look at that and try to find what the problem is with this? If you want to do a standalone generation of the edm4hep datamodel, the invocation of the code generator is:

python podio_class_generator.py <path/to/>edm4hep.yaml <path/to/output/> edm4hep ROOT

soumilbaldota · 2022-08-29T09:30:09Z

Hi @soumilbaldota, this looks like it solves things for the example datalayout of podio. I briefly tried to run it on the edm4hep definition and I have run into the following problem:

julia> include("edm4hep.jl")
WARNING: replacing module Vector3fModule.
WARNING: replacing module Vector3fModule.
[... more of those ...]
WARNING: ignoring conflicting import of Vector3dModule.Vector3d into Main
ERROR: LoadError: UndefVarError: ObjectID not defined
Stacktrace:
 [1] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/TrackerHitStruct.jl:5
 [2] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [3] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/TrackerHit.jl:1
 [4] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [5] top-level scope
   @ ~/work/EDM4hep/edm4hep/edm4hep/edm4hep.jl:17
 [6] include(fname::String)
   @ Base.MainInclude ./client.jl:451
 [7] top-level scope
   @ REPL[1]:1
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/TrackerHitStruct.jl:5
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/TrackerHit.jl:1
in expression starting at /home/tmadlener/work/EDM4hep/edm4hep/edm4hep/edm4hep.jl:17

From a glance at the stacktrace it looks as if some of the necessary includes are missing for edm4hep. Could you have a look at that and try to find what the problem is with this? If you want to do a standalone generation of the edm4hep datamodel, the invocation of the code generator is:

python podio_class_generator.py <path/to/>edm4hep.yaml <path/to/output/> edm4hep ROOT

Just fixed it.

tmadlener · 2022-08-31T12:13:35Z

I tried again with edm4hep, and it looks like the collections are not part of the module that is generated? Is that possible?

julia> include("edm4hep.jl")
julia> edm4hep.MCParticle()
MCParticleStruct{MCParticleStruct}(0, 0, 0, 0.0f0, 0.0f0, 0.0, Vector3dStruct(0.0, 0.0, 0.0), Vector3dStruct(0.0, 0.0, 0.0), Vector3fStruct(0.0f0, 0.0f0, 0.0f0), Vector3fStruct(0.0f0, 0.0f0, 0.0f0), Vector3fStruct(0.0f0, 0.0f0, 0.0f0), Vector2iStruct(0, 0), MCParticleStruct[], MCParticleStruct[])
julia> edm4hep.MCParticleCollection()
ERROR: UndefVarError: MCParticleCollection not defined
Stacktrace:
 [1] getproperty(x::Module, f::Symbol)
   @ Base ./Base.jl:35
 [2] top-level scope
   @ REPL[5]:1

Would it be possible to place the collections into the modules as well?

soumilbaldota · 2022-09-01T06:30:23Z

Hi @tmadlener, add Collection to namespaces fixes the problem now.

tmadlener · 2022-09-01T07:03:25Z

Very nice. Works for me now as intended. If you have any very clever idea of how to get rid of the multiple imports of the same components into the same module we can put that into a separate pull request.

* grouping code into modules * avoiding name conflicts between Structs and Constructor, including struct in main namespace only * adding parent namespace code generation * add Collection to namespaces

soumilbaldota added 2 commits August 21, 2022 23:29

grouping code into modules

0995757

fix pylint errors

b4de089

avoiding name conflicts between Structs and Constructor, including st…

42d89ec

…ruct in main namespace only

soumilbaldota changed the title ~~grouping code into modules~~ adding namespaces to Julia Aug 23, 2022

tmadlener reviewed Aug 24, 2022

View reviewed changes

fix accidentally commited files, add macro for julia_parameters

367bf50

tmadlener reviewed Aug 25, 2022

View reviewed changes

python/templates/JuliaCollection.jl.jinja2 Outdated Show resolved Hide resolved

soumilbaldota and others added 2 commits August 26, 2022 20:39

adding parent namespace code generation

cde25bb

Update python/templates/JuliaCollection.jl.jinja2

c5674bd

Co-authored-by: Thomas Madlener <thomas.madlener@desy.de>

tmadlener reviewed Aug 26, 2022

View reviewed changes

python/podio_class_generator.py Outdated Show resolved Hide resolved

soumilbaldota and others added 3 commits August 27, 2022 16:16

Update python/podio_class_generator.py

1995da5

Co-authored-by: Thomas Madlener <thomas.madlener@desy.de>

fix pyunittest, refactor datatype_component in namespaces

d8b6b48

fix pyunittest

0e07f68

fix missing includes

50c83a5

soumilbaldota added 2 commits September 1, 2022 11:51

add Collection to namespaces

e2b230d

fix flake8

d8ba97e

tmadlener approved these changes Sep 1, 2022

View reviewed changes

tmadlener merged commit 59fe1e7 into AIDASoft:julia Sep 1, 2022

soumilbaldota changed the title ~~adding namespaces to Julia~~ adding namespaces Sep 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

adding namespaces #325

adding namespaces #325

soumilbaldota commented Aug 21, 2022

tmadlener commented Aug 23, 2022

soumilbaldota commented Aug 23, 2022

tmadlener commented Aug 24, 2022

tmadlener left a comment

tmadlener Aug 24, 2022

tmadlener Aug 24, 2022

soumilbaldota Aug 24, 2022

tmadlener Aug 24, 2022

soumilbaldota Aug 24, 2022 •

edited

Loading

tmadlener Aug 24, 2022

soumilbaldota Aug 24, 2022

tmadlener commented Aug 29, 2022

soumilbaldota commented Aug 29, 2022

tmadlener commented Aug 31, 2022

soumilbaldota commented Sep 1, 2022

tmadlener commented Sep 1, 2022

		@@ -0,0 +1,2 @@
		include("{{ class.bare_type }}.jl")
		{{ class.bare_type }}Collection = Vector{ {{ class.bare_type }}{% if params_jl %}{ {% for relation in params_jl %} {{ relation }}, {% endfor %} }{% endif %} }

		include("datamodel/ExampleMC.jl")
		using .ExampleMCModule: ExampleMC

adding namespaces #325

adding namespaces #325

Conversation

soumilbaldota commented Aug 21, 2022

tmadlener commented Aug 23, 2022

soumilbaldota commented Aug 23, 2022

tmadlener commented Aug 24, 2022

tmadlener left a comment

Choose a reason for hiding this comment

tmadlener Aug 24, 2022

Choose a reason for hiding this comment

tmadlener Aug 24, 2022

Choose a reason for hiding this comment

soumilbaldota Aug 24, 2022

Choose a reason for hiding this comment

tmadlener Aug 24, 2022

Choose a reason for hiding this comment

soumilbaldota Aug 24, 2022 • edited Loading

Choose a reason for hiding this comment

tmadlener Aug 24, 2022

Choose a reason for hiding this comment

soumilbaldota Aug 24, 2022

Choose a reason for hiding this comment

tmadlener commented Aug 29, 2022

soumilbaldota commented Aug 29, 2022

tmadlener commented Aug 31, 2022

soumilbaldota commented Sep 1, 2022

tmadlener commented Sep 1, 2022

soumilbaldota Aug 24, 2022 •

edited

Loading