Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

New working item for BigTIFF community standard #103

Open
joanma747 opened this issue Aug 24, 2021 · 28 comments
Open

New working item for BigTIFF community standard #103

joanma747 opened this issue Aug 24, 2021 · 28 comments
Labels
done

Comments

@joanma747
Copy link
Contributor

joanma747 commented Aug 24, 2021 •
edited
Loading

BigTIFF has become more necessary that ever with lager images, multiband images and multiple resolutions (COG).
The Testbed 17 has a proposal for a new draft specification for BigTIFF: https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/tree/master/standardDrafts/BigTIFF

We would like to move it to an OGC community standard in this group. The process is a bit different form other community standards. There is only an "informal" web page that defines BigTIFF. IT is clear and comprehensible but not sufficiently formal, so we cannot adopt any preexisting document directly. It was needed to reformatted it into clear requirements.

The issue is identified in #12 where an analysis of the implications for BigTIFF tags in GeoTIFF keys could have is presented
@joanma747 joanma747 changed the title New working item for BigTIFF standard New working item for BigTIFF community standard Aug 24, 2021
@ogcscotts
Copy link
Contributor

@joanma747 - two questions on the Community Standard.

  1. Do the owners of BigTIFF agree to submit the document?
  2. If the Community Standard is being written as a result of Testbed activity, we will need to get the owner of BigTIFF to agree to the text and that it represents an authoritative specification for BigTIFF before we could consider this for a Community Standard. It may also be worth considering whether the BigTIFF ownership could be transferred to OGC.

@cmheazel
Copy link
Contributor

An alternate approach, add a BigTIFF conformance class where we document the differences between TIFF and BigTIFF. No need for an external standard.

@joanma747
Copy link
Contributor Author

joanma747 commented Aug 24, 2021 •
edited
Loading

Let's advertise our initiative in the LibTIFF email list and look for their support. We should strongly underline that we are fully compatible with the previous documentation and implementation of the BigTIFF.

Consider to make BigTIFF a public repo

MOTION to establish an OGC BigTIFF standards development effort, using a public Git (gitlab or github) repository and invite participation of the BigTIFF community
Moved by Chuck Heazel
Second by Even Rouault
Discussion: We are going to use the LibTIFF email list. IT seems that it will be and OGC standard with support of the community. It is not clear if this is an "OGC community standard" or a "pure OGC standard with the support of the community" at this point.
NOTUC

We will ask Greg for a new repository for this working item. It makes configuration management easier

Let's draft some simple text to do the first contact to the LibTIFF list.

@cmheazel, the requirement class for BigTIFF is here:
https://gitlab.ogc.org/ogc/T17-D046-COG-Specification-ER/-/blob/master/standardDrafts/BigTIFF/standard/clause_7_normative_text.adoc

@joanma747
Copy link
Contributor Author

The repository to work in this working item is here:
https://github.com/opengeospatial/BigTIFF

@joanma747
Copy link
Contributor Author

The repository is now populated with the materials created from Testbed17. The intention is to keep them aligned (manually) until the end of Testbed 17, when the work will exclusively continue in the GeoTIFF.SWG

@joanma747 joanma747 added the done label Sep 7, 2021
@joanma747
Copy link
Contributor Author

joanma747 commented Sep 28, 2021 •
edited
Loading

Proposal of text email for the LibTIFF list:

A group of people in the Open Geospatial Consortium (OGC, www.ogc.org) would like to move the BigTIFF into the OGC standards process. The intention is to create a OGC standard that is fully compatible with the previous documentation and implementation of the BigTIFF. We would like to do this in a open process and make sure that the community that was behind the BigTIFF specification are invited to participate in the process and are well recognized for their previous work. The OGC will provide a process to maintain the standard in the future and will work to make GeoTIFF and Cloud Optimized GeoTIFF (COG) standards to support both TIFF and BigTIFF. All OGC standards are available for free in the OGC website.

The process will be done in a open GitHub repository and the first attempt can be found here: https://github.com/opengeospatial/BigTIFF
You are invited to submit issues or to create pull requests as part of the way to provided comments and improve the standard draft.

Is there any objection or comment before we start this process?

@joanma747
Copy link
Contributor Author

Even Rouault will send this to the LibTIFF email list. Thanks for that!

@rouault
Copy link
Contributor

rouault commented Sep 28, 2021

email just sent

@rouault
Copy link
Contributor

rouault commented Sep 28, 2021

Good question from Rob Tillaart regaring TIFF tags in general:

Rob Tillaart: Will the OGC put a process in place for BigTiff tags?

me: > What do you call exactly as BigTIFF tags ?

Rob Tillaart: In the past there was a lot of discussion about (un)documented / (un)registered tags and how to handle them.
Will the OGC play a role in this process of documenting tags?
I can imagine that they only are interested in Geospatial related tags and not in e.g. medical tags

@rouault
Copy link
Contributor

rouault commented Sep 28, 2021

"""
Hi Even et al:

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems).
The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica.
The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list, and documenting essentially a copy of Ole's work. I don't believe the participants were from Aware.
Anyone do please correct me if my understanding is incomplete or incorrect.

However, this unofficial specification only covers a migration of TIFF to 64-bits, and says nothing in particular about tiling, multi-resolutions, multi-channels, bit depths, compression, TIFFtags/metadata, etc, all of which need to be defined per application at a higher level.
Various stabs at this for multiresolution imaging have been made over the years: Aperio.svs, COG, ZIF (my own), OME-TIFF (used ZIF as a reference point), and many 99% similar flavors of pyramidal tiled (big)TIFF. Shame we can't all get it together.

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

There is no formal maintainer; the AWARE systems site does not maintain the BigTIFF specification, that is just a reference; another is at http://bigtiff.org, a site now owned by Leica.

And I'm not sure the wording on https://github.com/opengeospatial/BigTIFF is as intended?: "it becomes an OGC standard" sounds like OGC intends to take over ownership of BigTIFF? Leica might have something to say about that, although the specification has been put in the public domain.
Personally, I really wish Adobe as the owner of the TIFF specification would step up to the plate with this, and update the actual TIFF specification to cover BigTIFF. Their focus has been elsewhere for many years, but this would be a great community service.

Regards,

Kemp Watson
Objective Pathology Services
Toronto, Canada
"""

@rouault
Copy link
Contributor

rouault commented Sep 28, 2021

Some clarification given by Joris Van Damme on the non-central implication of Aperio in the BigTIFF design:
"""
Kemp,

To my knowledge, the community that was behind the BigTIFF specification is now effectively Leica (medical, not Geosystems).

Incorrect. Leica, as a company, was only involved in that they were
one of four sponsors of the BigTIFF implementation inside LibTIFF.
They were completely unrelated to the design. It's possible a number
of people that contributed to the design in the mailing list were
employed by Leica at the time, but if so they were not speaking for
that company.

The initial specification came from Ole Eichhorn, then at Aperio, a digital pathology company which was acquired by Leica.

No. Original specification was a community effort, on the mailing
list, and to this day you can read up on that and verify it in the
mailing list archive. Aperio, at that time, had an alternative design,
partly incompatible, that they put into an independent implementation,
without consulting the community. It made things quite awkward for a
while. They still own bigtiff.org. I hope that at least the stuff they
offer on that site is no longer incompatible with our own
specifications, but I haven't checked.

The involvement of Aware systems as far as I know was only hosting a discussion on their mailing list

Incorrect. AWare Systems is me, Joris Van Damme. I actively
contributed to the design phase quite a bit, though it truly was a
community effort. I also did the implementation in LibTIFF, sponsored
by four other companies.

Various stabs at this for multiresolution imaging have been made over the years

The SubIFDs tag allows for any number of any resolution child ifds.
It's a specification totally independent from BigTIFF, SubIFDs can and
should be used in classic TIFF and BigTIFF. You're not making sense.

I can go on correcting your other statements, but I fail to see how
that's worth the effort at this point. It's ancient history. Why spin
these things, what is your angle?

Given that GeoTIFF and COG specifications exist, what is OGC intending to add beyond that? Is it just the "64-bit"ness, or metadata?

Like the man said, it is the BigTIFF part, so, sure, the "64-bit"ness.
BigTIFF is just TIFF with 64bit offsets, by design.

I feel the current activity in the GeoTIFF community is a good thing
that will surely result in more widespread BigTIFF adoption, and more
widespread confidence in its validity and the fact that it's here to
stay. Even, thanks for letting us know.

Best regards,

Joris Van Damme
AWare Systems
"""

@joanma747
Copy link
Contributor Author

It is so great that we have such support from Joris Van Damme. I love the tone of the last paragraph. This email clarifies the history and helps a lot.

@rouault
Copy link
Contributor

rouault commented Sep 29, 2021 •
edited
Loading

Project of response to the reactions from the libtiff mailing list

Hi,
thanks to all who have provided feedback. Our group just met and we'd like to provide the following clarifications:
- we'll change the readme of the github repository to mention that the current BigTIFF specification is hosted on the awaresystems.be website rather than maintained.
- we do not plan, at least at that stage, to maintain a full TIFF tag registry. OGC only intends to maintain the GeoTIFF tags, as well as tags for geospatial metadata.
- the scope of the BigTIFF specification maintained by our OGC group will be exactly the same as the one mentioned above. As you may see from the draft, it is a relatively short specification, documenting only the differences between BigTIFF and classic TIFF, that is the evolution of the header, the directory structure, and more generally all the 64-bits indexing / offsets extension of TIFF6 specification. The TIFF 6 spec will be the reference for all aspects not modified by BigTIFF.
- "who is responsible for BigTIFF?" That's indeed the main issue that triggered our initiative. Our review of the current situation was that no one was formally a "owner" or in charge, or showing hints to be willing to be in charge, hence we've decided to step up. Our motivation to do that is clearly to be able to use BigTIFF in a GeoTIFF context. But BigTIFF having not a formal enough specification, this makes it hard to comply with the requirements of a standards process. We've considered to put BigTIFF as an annex of the GeoTIFF specification, but that would not have been a clean approach. We believe that by making BigTIFF a standalone specification, it might also be useful for other communities, beyond the geospatial one, and will make further adoption of the technology easier in contexts where a formal specification is needed.

Can we answer if there will be an ETS for BigTIFF ?

@EmDevys
Copy link
Contributor

EmDevys commented Sep 29, 2021

@rouault (and all)
2 comments:

  • "we do not plan, at least at that stage, to maintain a full TIFF tag registry": I would add (for clarification) that: the OGC only intends to maintain the GeoTIFF tags, as well as tags for geospatial metadata.
  • "the scope of the BigTIFF specification maintained by our OGC group will be exactly the same as the one mentioned above." I would add something here for clarification, such as "the "64-bit"ness capability", or the 64-bits indexing / offsets extension of TIFF6 specification.

@joanma747
Copy link
Contributor Author

I agree with the answers @rouault provides with the good additions that @EmDevys suggests.

@rouault
Copy link
Contributor

rouault commented Sep 29, 2021

I've integrated Emmanuel's suggestions. Should we wait next week meeting to validate the message ? (that should be fine)

@rouault
Copy link
Contributor

rouault commented Sep 29, 2021

The whole discussion on the libtiff mailing list is archived at https://www.asmail.be/msg0055229449.html

@joanma747
Copy link
Contributor Author

joanma747 commented Sep 30, 2021 •
edited
Loading

De : Scott Foshee [mailto:sfoshee@adobe.com]
Envoyé : lundi 26 octobre 2020 21:53
À : Emmanuel Devys; Scott Simmons (ssimmons@ogc.org)
Cc : wprotzman@dcscorp.com; 'Wilkes, Graham (NRCAN/RNCAN)' (graham.wilkes@canada.ca); Jennifer Hum-Miller; Roger Lott (EPSG); Even Rouault (even.rouault@spatialys.com); Charles.M.Heazel.ctr@nga.mil; Larry Beck; geotiff.swg@lists.opengeospatial.org; Scott Foshee
Objet : Re: Adobe is in full control of the TIFF standard

Hi Emmanual and Scott…

Since Adobe is the copyright holder for Adobe TIFF, you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.

I explained that this was not an option in an earlier e-mail.

I would be happy to convey this directly to the OGC legal department. Scott, are you the correct person at OGC for Adobe to convey copyright issues?

Also, I know the folks at ISO and will update them.

Thanks,

Scott

Where this boom comes from?. If I interpret this text literally (you can not create a Geospatial TIFF standard that’s based on Adobe TIFF without a license from Adobe.), this could affect also COG as well as GeoTIFF itself!. That could be the end of any of these initiatives (except if Adobe grants a licence to the OGC for free). Not everybody is for open standards. Lets hope that "OGC diplomacy" can navigate the situation.

@EmDevys
Copy link
Contributor

EmDevys commented Sep 30, 2021 via email

@joanma747
Copy link
Contributor Author

I understand your points with the GeoTIFF and I'm sure all was considered at the time.

The question now is if this is red flag for our activities in BigTIFF and COG. Can we assume that a similar approach will, in the end, work and continue the technical work?. I hope so!

@rouault
Copy link
Contributor

rouault commented Sep 30, 2021

The question now is if this is red flag for our activities in BigTIFF and COG. Can we assume that a similar approach will, in the end, work and continue the technical work?. I hope so!

My opinion: Adobe is just throwing FUD (Fear Uncertainty Doubt) to try to extract a few more $$$ from their 1994's Aldus acquisition, and they don't own any copyright or trademark on BigTIFF, nor as they do on GeoTIFF.

@EmDevys
Copy link
Contributor

EmDevys commented Sep 30, 2021 via email

@ogcscotts
Copy link
Contributor

We seem to have a contact at Adobe again who is engaged in the discussion - OGC staff will see if we can get more firm legal insight from Adobe.

@joanma747
Copy link
Contributor Author

Is this settling the issue for good?. It looks like it does!!!

Leonard Rosenthol via Tiff tiff@lists.osgeo.org
Oct 4, 2021, 9:14 PM (11 hours ago)
to Emmanuel, Kemp, Joris, geotiff.swg@lists.opengeospatial.org, Scott, tiff@lists.osgeo.org

Scott Foshee (who manages all imaging standards for Adobe) and I spoke today and we would like the community to know that we are supportive of BigTIFF being moved from its current state into a formal open international standard by building on top of an already existing open standard for TIFF – that being TIFF/EP (https://www.iso.org/standard/29377.html). We would recommend that the work take place at ISO as well, so that others can then build on top of it as they have with TIFF/EP itself, and would also make it easier for Adobe to participate (if that is desirable by the community). However, we understand if the OGC or others would prefer some other formal SDO.

I will note that this is response is ONLY about BigTIFF and does not have any impact on any other conversations around GeoTIFF or others that may be ongoing.

Leonard

@EmDevys
Copy link
Contributor

EmDevys commented Oct 5, 2021 via email

@joanma747
Copy link
Contributor Author

joanma747 commented Oct 5, 2021 •
edited
Loading

Transcribing the discussion of the today telecon of the SWG:

We seem to have two requirements emerging from the discussion:

  • It should be freely available
  • Publish it in ISO

There are many precedents of standards done by the OGC that comply with both requirements. We propose to start in the OGC and propose it to ISO using JAG or a similar mechanism.
There are concerns on how much extra work going to ISO this will require. It will require a country to propose it as a working item.

We need to start a discussion with the OGC staff about the best way to label the document and bring it to the ISO process (e.g. usign the JAG).

@EmDevys
Copy link
Contributor

EmDevys commented Oct 5, 2021 via email

@desruisseaux
Copy link
Contributor

This issue is labelled as "done". Is this label accurate? From my reading of above discussion, the legal question seems to still open. If this issue is indeed done, can we close this issue with a comment summarizing the outcome?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
done
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@rouault @desruisseaux @joanma747 @cmheazel @EmDevys @ogcscotts