Remove duplicate netcdf entry (triggers "Type application/netcdf is already registered as a variant of application/netcdf.")

[shioyama]

This seems to be a duplicate entry. We've encountered an issue where the second entry triggers a warning here:

https://github.com/mime-types/ruby-mime-types/blob/5b931d70d74f3d1ba25babac03237ae450ca5c5b/lib/mime/types.rb#L189-L193

    if !quiet && @type_variants[type.simplified].include?(type)
      MIME::Types.logger.warn <<-WARNING.chomp.strip
        Type #{type} is already registered as a variant of #{type.simplified}.
      WARNING
    end

This emits Type application/netcdf is already registered as a variant of application/netcdf.

I've just removed the first of the two. The change traces back to 8c995e61 released in v3.2022.0105.

---

This change is 

[halostatue]

Please check types/application.yml and types/provisional-standard-types.yml and modify those. Simply modifying the data/mime.content_type.column file will break everything (there are 8 column files). The files in data/ are generated files and should not be modified manually. The source will be in one or both of the above files.

[shioyama]

@halostatue I've updated types/provisional-standard-types.yml and run rake convert:yaml:json and rake convert:yaml:columnar to generate data files.

[halostatue]

In general, PRs should just be for the YAML files; I do the conversion (there are three conversions now) as part of the release process. It’s not a big deal.

[halostatue]

So I’ve done a quick bit of checking, and the story isn’t great at this point.

Removing the type from the provisional file won’t do any good, because there’s a provisional registration with IANA, so the next run of rake mime:iana will restore the provisional record.

Removing it from types/application.yml isn’t great either, because while it’s not registered, it does have some extensions recorded:

- !ruby/object:MIME::Type
  content-type: application/netcdf
  encoding: base64
  extensions:
  - nc
  - cdf
  registered: false

I deliberately do not permit the customization of the provisional file, and we don’t currently overlay the provisional data on the existing types files, so we either:

1. continue with this warning as is,
2. remove the unregistered application/netcdf entry from types/application.yml (and lose the extensions), or
3. change how provisional types are handled from the IANA converter such that they overlay any types in the primary types.

We would need to probably keep the provisional file so that we can see when types are removed from provisional registration without being promoted to official registration (maybe we keep those but mark them as registered: false).

The third option is the correct option, but it is also a lot of code change in the support code because it would now need to deal with promotion and/or removal. I also don’t know how important the extensions are for application/netcdf, so I can’t speak on the harm that option 2 would cause; it could be mitigated by adding an issue wanting the extensions added back once application/netcdf is a properly registered (and not merely provisional) type.

Unfortunately, the means that this PR cannot be merged as is. I’m not closing it yet, as we need to figure out the correct resolution.

[shioyama]

Thanks for the very detailed response! The warning may confuse people, but it's not urgent. We've opted to just add a line that ignores it, which works for us until the issue is resolved.

I'll let you decide if/when to close this, and thanks again for the explanation.

[noff]

Any updates on this issue? The warning is quite annoying due to it outputs the message to STDOUT and cron MAILTO command delivers it as output of the cron task.

[halostatue]

Not at the moment, sorry. It’s not a trivial issue to resolve. If you’d like to open an issue on mime-types/ruby-mime-types, I can look at ways that this message can be suppressed (it won’t be removed, but I could suppress it either with an environment variable or by checking a common environment variable like RAILS_ENV), but I have an extremely busy schedule this week so it will probably not be looked at until the weekend or next week.

[dogweather]

IMO the problem is, the library shows a warning message to the wrong audience (application developers). It's not understandable to app devs, not marked with its provenance (source lib name), and finally, app devs have no way to resolve it.

Instead, the message seems aimed at mime-types-data library developers. So, what's the shortest path to changing the log level (DEBUG ?) and/or logic to show it only to the correct audiences?

I submitted a PR to make this DEBUG level: mime-types/ruby-mime-types#170

[halostatue]

IMO the problem is, the library shows a warning message to the wrong audience (application developers). It's not understandable to app devs, not marked with its provenance (source lib name), and finally, app devs have no way to resolve it.

This is incorrect. The message is to developers who use mime-types. Yes, a lot of people only use mime-types for the associated data (mime-types-data), but it is perfectly extensible, and this warning is completely correct in that case.

[dogweather]

Yes, a lot of people only use mime-types for the associated data (mime-types-data)...

How can those people be better supported?

[halostatue]

Yes, a lot of people only use mime-types for the associated data (mime-types-data)...
How can those people be better supported?

I don’t have an answer for that—as the lack of resolution to this PR and other possible fixes shows. It is problematic when multiple matching types are added, so the warning isn’t inappropriate. There’s no way to say "ignore any warnings while loading this, but then start showing warnings after this is loaded". Even if there were, I’m not sure whether that would be the right thing to do, because there’s a more fundamental issue of duplicate data in this database, and it appears to be unavoidable with the design constraints currently present.

[dave105010]

This situation is creating a lot of clutter in our logs. It started appearing after Rails 7 upgrade. This also affects our cron scripts that run periodically. All of them are sending the warning output and notifying us. Is there a way to get rid of this warning?

[halostatue]

I have addressed this properly in #53 and will be releasing a new version of mime-types-data tonight that resolves this issue by merging provisional types into the main types and not handling them separately.

Thanks for your contribution, @shioyama and my apologies for how long it took to resolve this.

[dave105010]

Thank you Austin!! On February 17, 2023 at 8:37:26 PM PST, Austin Ziegler ***@***.***> wrote: !! External Email I have addressed this properly in #53<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmime-types%2Fmime-types-data%2Fpull%2F53&data=05%7C01%7Cdaveoz%40vmware.com%7Ce3f3f5132fb644bbc8a508db11694c2d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638122918464269535%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fTumnTNF4wFtxjQR3kylLitMToaKiFUMdW8u%2Bw%2FjHNQ%3D&reserved=0> and will be releasing a new version of mime-types-data tonight that resolves this issue by merging provisional types into the main types and not handling them separately. Thanks for your contribution, @shioyama<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fshioyama&data=05%7C01%7Cdaveoz%40vmware.com%7Ce3f3f5132fb644bbc8a508db11694c2d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638122918464424577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=0DlKs6sfrF96t4R4ImvoKvay46u%2FlLzGLSuK0LvM7sk%3D&reserved=0> and my apologies for how long it took to resolve this. — Reply to this email directly, view it on GitHub<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmime-types%2Fmime-types-data%2Fpull%2F50%23issuecomment-1435480548&data=05%7C01%7Cdaveoz%40vmware.com%7Ce3f3f5132fb644bbc8a508db11694c2d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638122918464424577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=JGahBzkDWRVxlvnEqWqVuWl8J%2BZaIievBqDj3ud8lHE%3D&reserved=0>, or unsubscribe<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAOTTVWMGW7CMB4LLPCMG4WDWYBGJ5ANCNFSM5MDZMGTA&data=05%7C01%7Cdaveoz%40vmware.com%7Ce3f3f5132fb644bbc8a508db11694c2d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C638122918464424577%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=TM%2F4Qy3O4suF%2FIP%2FHNhvSyCKFd%2FvTRQckbkIGMP22%2Bo%3D&reserved=0>. You are receiving this because you commented.Message ID: ***@***.***> !! External Email: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender.